[Ingest] Ingest Node cannot find pipeline that exists #28269
Description
I ran into an issue, which I cannot reproduce, where a set of ingest nodes were complaining that they could not find an ingest pipeline during an ordinary _bulk request that used it. This followed a single node cluster being upgraded from 6.1.1 to 6.1.2 by adding a new node, then decommissioning the old node. While trying to debug the issue, two other nodes were added, which were also ingest nodes.
I was able to see the Ingest pipeline by requesting it directly:
GET /_ingest/pipeline/xpack_monitoring_6as well as via the cluster state:
GET /_cluster/state/metadata?filter_path=metadata.ingest.pipelineIn both cases, it returned the expected ingest pipeline:
{
"xpack_monitoring_6": {
"description": "This is a placeholder pipeline for Monitoring API version 6 so that future versions may fix breaking changes.",
"version": 6010099,
"processors": []
}
}Workaround
Fixing the issue was simple, but it's unclear what was actually wrong. Simply delete, then replace the exact same ingest pipeline with the same name (it's possible that the DELETE step is unnecessary):
DELETE /_ingest/pipeline/xpack_monitoring_6
PUT /_ingest/pipeline/xpack_monitoring_6
{
"description": "This is a placeholder pipeline for Monitoring API version 6 so that future versions may fix breaking changes.",
"version": 6010099,
"processors": []
}From that point onward it began to work.