5.0 Default pipelines

[niemyjski]

The more I use pipelines the more useful they would be come if I could specify a list of pipelines that automatically get run on a type or index level.

This would also save some overhead of specifying what pipeline to use when a huge percentage of use cases that are using pipelines will never change. This would also make the api easier to use.

Some questions would be, if I specify a list of pipelines on an index, what would happen if I specify a pipeline to use, Would it be a merged list or just the specified pipeline to run.

[clintongormley]

The more I use pipelines the more useful they would be come if I could specify a list of pipelines that automatically get run on a type or index level.

I've been against default pipelines because pipelines should only be run on the first ingestion. When you update or overwrite a document, you may not want the default to run. For this reason I prefer pipelines to be manually specified.

[djschny]

With _timestamp removed if a user wants to add their own timestamp field, a pipeline processor is really the only way to do it. Having to force all clients to specify the same pipeline (or include it in theirs) is problematic. I reached for this within the first 30 minutes of using pipelines and feel it would be very helpful.

[inqueue]

When you update or overwrite a document, you may not want the default to run.

I agree for a non-logging use case. The ability to enable default pipelines for a logging use case would be very helpful where document updates are non-existent. Further, I would only want to enable default pipelines for certain indices or have the capability to do so. Could it be an index setting?

Example uses for logging might be password stripping with the set processor or field truncation using a script processor.

[tipuban]

+1 to allowing default pipelines for indices.
Ideally you would also be able to specify on what type of operation the pipeline would be run: insert-time, update or both.

[cristimagda]

Ideally you would also be able to specify on what type of operation the pipeline would be run: insert-time, update or both.

+1 It would be very useful.

[marius-dr]

+1 for this. It helps users that were affected by the removal of _timestamp field.

[zoellner]

another argument in favor of being able to specify a default pipeline:
There are scenarios where the PUT command (and some other document pre processing is outside the control of the ES operator)

AWS allows to feed an elasticsearch instance from a Amazon Kinesis Firehose Stream. However, the document _id is set by the Firehose Stream. Firehose also controls the command that is used to send the data to the elasticsearch instance, i.e. it is not possible to add the ?pipeline query parameter.
With a default ingest pipeline (based on index/type, ideally specified altogether in the index template) one could set the _id through a preprocessor based on the document _source.

[titzi]

+1 especially as an option block in index templates, I guess this would be a perfect spot for it.

[whiteboardmonk]

+1 having this in the index template will be very useful

AWS allows to feed an elasticsearch instance from a Amazon Kinesis Firehose Stream. However, the document _id is set by the Firehose Stream. Firehose also controls the command that is used to send the data to the elasticsearch instance, i.e. it is not possible to add the ?pipeline query parameter.
With a default ingest pipeline (based on index/type, ideally specified altogether in the index template) one could set the _id through a preprocessor based on the document _source.

@zoellner I stumbled upon this issue while looking for the exact same option. Did you happen to figure out a way around this?

[zoellner]

@whiteboardmonk no, I've since stopped using Firehose Streams because of this issue.

[wpongra]

+1, _timestamp-replacement as the use-case

[niemyjski]

Is there any updates on this?

[mr-mos]

+1, for the default pipeline (regarding the use-case for _timestamp)