Support Query DSL metadata

[PhaedrusTheGreek]

It would be convenient if the query DSL had some metadata component, so that we could tag queries. This would be useful in the case where certain queries are identified as being problematic (such as in the slowlog), and the origin in unknown. e.g., if Kibana could tag a query with the name of the visualization, it would make it simple to ID when there are, say, hundreds of visualizations.

GET /traffic-logs/_search
{
   "query": {
   },
   "size": 0,
   "meta": {
        "kibana_instance_id": "some_id"
        "kibana_dashboard_label": "My Deep Aggregations"
        "kibana_vis_label": "Error Codes by Country"
   },
   "aggs": {
   }
}

[s1monw]

I think these information would be useful, yet I think we should not arbitrary headers here. I also think we shouldn't extend the query DSL for this maybe we can use http headers for this which have to be explicitly enabled to be transported further. We can have for instance a set default headers like es-client-id and / or es-request-desc those are then automatically avaliable throughout the entire lifecycle of the request on any API.

[PhaedrusTheGreek]

+1 for http headers @s1monw . I also wonder how we could tie a Shield audit log entry to a slowlog entry, to track source and user, etc.

[geekpete]

I thought something similar to this idea would be super useful as well.

My example use case is where a user has 1000 visualisations in Kibana and one or more is causing the cluster issues with the queries it produces. So the ability to tag a query so that the tag remains you do a heap dump and pull the problem query out of memory would be invaluable in troubleshooting and pinpointing the original source of a query. Starting with Kibana visualisation identification but also presenting this as a functionality that end users can utilise to tag their own queries.

I just need a single metadata field as a label/tag that is essentially ignored by everything (except maybe some metrics collection).

It also might be important to consider any caching mechanisms here, is it best for caching to ensure it always ignores the metadata "tag" in queries in the scenario of the same query hitting the cluster but with many different tags. Or would tag-aware caching provide benefits as well? Blended/optional/configurable caching approaches?

[siddharthlatest]

Has this been implemented? A use-case I am looking at requires the response from the latest request to be rendered in the UI. Since server processing times may vary, and in my case, the latter timed request is usually a more pruned aggregation often resulting in quicker processing than the former request's more broader aggregation. In this scenario, being able to add timestamp as a meta-data would be a nifty way to resolve which response's hits should be rendered.

[jasontedor]

@siddharthlatest The issue is open so no, it has not been implemented.

[nik9000]

@imotov, now that we have x-opaque-id I wonder if we can close this? Should we wire it into the slow logs first? Into the audit log too?

[imotov]

@nik9000 yes, I think we can probably address this by adding the id to logs. We can either repurpose this issue or close it and open a new one.

[tomcallahan]

Closed in favor of #31521