Ingest Pipelines not working after removing plugin

[gmoskovicz]

Elasticsearch version (bin/elasticsearch --version): 6.0

Plugins installed: ingest-geoip

JVM version (java -version): Any

OS version (uname -a if on a Unix-like system): Any

Steps to reproduce:

1. Install ingest-geoip plugin in Elasticsearch
2. Create a pipeline like the one that shows here: https://www.elastic.co/guide/en/elasticsearch/plugins/6.0/using-ingest-geoip.html
3. Create another simple pipeline [1]
4. Stop Elasticsearch
5. Remove the ingest-geoip plugin in Elasticsearch
6. Start Elasticsearch
7. Try to index a document [2]

[1]

PUT _ingest/pipeline/my-pipeline-id
{
  "description" : "describe pipeline",
  "processors" : [
    {
      "set" : {
        "field": "foo",
        "value": "bar"
      }
    }
  ]
}

[2]

POST index/test?pipeline=my-pipeline-id
{
  "something": true
}

Elasticsearch will return:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "pipeline with id [my-pipeline-id] does not exist"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "pipeline with id [my-pipeline-id] does not exist"
  },
  "status": 400
}

It seems to be that the problem is coming from the beginning when trying to apply the cluster state:

[2017-12-12T14:44:24,067][WARN ][o.e.c.s.ClusterApplierService] [_vjNi7c] failed to notify ClusterStateApplier
org.elasticsearch.ElasticsearchParseException: No processor type exists with name [geoip]
	at org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException(ConfigurationUtils.java:239) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:341) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:259) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.ingest.Pipeline$Factory.create(Pipeline.java:122) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.ingest.PipelineStore.innerUpdatePipelines(PipelineStore.java:82) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.ingest.PipelineStore.applyClusterState(PipelineStore.java:69) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:495) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.cluster.service.ClusterApplierService$$Lambda$2050/1070007033.accept(Unknown Source) [elasticsearch-6.0.0.jar:6.0.0]
	at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_45]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:492) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:479) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:429) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:158) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-6.0.0.jar:6.0.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]

[talevy]

@gmoskovicz so is there a specific behavior that you prefer to see?

One thing that comes to mind to me is that we continue to throw an exception in the logs, but also continue to registering later pipelines in the loop that would be able to be registered. Does that sound fair? So, pipelines that did rely on those plugins would still not be present, but the others would still work.

[gmoskovicz]

Do we need to throw an intermittent exception? Perhaps just at startup (or when it fails) is enough?

[rjernst]

@talevy +1 to your plan.

@gmoskovicz What do you mean by "intermittent exception"? Node startup is exactly what is being discussed. There is no synchronous request to return an error in here.

[gmoskovicz]

Ah, ok. +1 on that then.

I saw some intermittent logging related to this issue, but it might be just related to this issue but not caused by the same facts. It's just present when xpack is installed since it uses a pipeline for the Monitoring API.

[talevy]

I know this is late to the game... but a similar problem popped up recently, and so that change should also fix this. reference PR:

#28752

Since we cannot stop the node from starting up, the best we can do is inform the user that there is a problem. the PR I have pushed should make it clear to users that attempt to run pipelines, why they failed. Other pipelines that did not depend on the geoip processor should still be loaded now and work