Cat endpoint for ingest pipelines

[markwalkom]

Describe the feature:
It's be great if there was a _cat endpoint that showed defined pipelines.

It might mean we need to save a bit more info into cluster state with each pipeline, perhaps things like created + updated times. I don't know if there's more metric based info we are planning for these, but things like processed/failed events (for eg) might also be useful.

[elasticmachine]

Pinging @elastic/es-core-infra

[rjernst]

We are discussing adding stats like numbers of processed/failed ingestions to existing stats apis. Given that the configured names are not generally useful from an operator point of view, I'm going to close this in favor of that future work.

[inqueue]

As the use of ingest pipelines picks up with Beats modules, so does the need to eventually purge older unused and unwanted pipelines. Having a _cat endpoint for pipelines could help with their discovery, like _cat/templates or _cat/aliases. GET /_ingest/pipelines from Kibana Dev Tools is ok, but it is only JSON output where sometimes only a list of pipelines is needed.

Consider an older cluster that was created with 5.6 that is now running 7.0. It would have these pipelines if it accepts data from the Filebeat Nginx module:

curl $CURL/_ingest/pipeline | jq '.|keys'
[
  "filebeat-5.6.15-nginx-access-default",
  "filebeat-5.6.15-nginx-error-pipeline",
  "filebeat-6.6.1-nginx-error-pipeline",
  "filebeat-6.6.2-nginx-access-default",
  "filebeat-6.6.2-nginx-error-pipeline",
  "filebeat-6.7.0-nginx-access-default",
  "filebeat-6.7.0-nginx-error-pipeline",
  "filebeat-7.0.0-rc1-nginx-access-default",
  "filebeat-7.0.0-rc1-nginx-error-pipeline",
  "xpack_monitoring_2",
  "xpack_monitoring_6",
  "xpack_monitoring_7"
]

That is 662 lines of JSON in the Dev Console. There are several pipelines here that need to be removed and most, if not all, could be unknown to the admin anyway. And this is just one Filebeat module. Other modules ship with their own pipelines, hence eventually some management is needed here (there is probably a separate issue needed for Beats).

Could we give this feature request another consideration?

cc @ruflin

[m9aertner]

A /_cat/pipelines endpoint would also be nice for consistency with /_cat/aliases and others.

When there is no pipeline defined, /_ingest/pipeline as shown above causes a 404 error, whereas 200 OK is returned for no aliases. I prefer the latter, as it's more convenient for fail-fast scripting.

Note that /_ingest/pipeline/* and /_ingest/pipeline/my-* also show 404.

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Ingest)