Ability to associate a search task ID

[eskibars]

Describe the feature:
When you fire off a search request to Elasticsearch, you're stuck waiting until the result comes back. Normally, that's very, very fast. But occasionally an egregious search/dataset can take a while to get through, so we added the ability to kill them through the task manager. That's great, but it's difficult to use from the UI that executes the search.

Consider:

1. User enters a search that will be slow into an external UI
2. UI executes search to Elasticsearch
3. User wants to cancel before results come back

How does the UI match up the search that was executed with the list of tasks that are in the system? The UI could try some heuristics to match up the search descriptions in the task manager to the original request, but it'd have to be done based upon heuristics and those heuristics become very difficult given Elasticsearch will have rewritten the query and that there could be multiple searches that match the heuristic.

It'd be nice if you could associate some ID with a request at search time and have that ID show up in the task manager. That way, when the UI executes the request, it could specify an ID it could reference later if it needs to kill the request.

[evanvolgas]

I assume you are talking about something like generating a unique request id along the lines of https://blog.ryandlane.com/2014/12/11/using-lua-in-nginx-for-unique-request-ids-and-millisecond-times-in-logs/. If so, I am hugely in favor of this idea, especially if the search id were carried through to the slow query logs. If it were, that would be extremely helpful vis a vis efforts around improving slow query logging (eg #9172 and #12187 (comment)). It could also potentially lend itself to @PhaedrusTheGreek's idea of breaking down API response time (#21073) or even logging it outside of just the Profile API.

[nik9000]

We've talked about this on and off for a while.

If we do this I think it'd be easier if this were a thing for tasks in general rather than just searches. It might work like task status. It is a general thing but each request has to "opt in" to it. There should be a "standard" way to opt into it.

I think it'd be hard if we wanted to force these IDs to be unique because we don't have a good place for that.

I'm thinking of a task metadata url parameter which could be search using the list tasks API. Or something like that. @imotov, what do you think?

[imotov]

@nik9000 maybe we can somehow expose a whitelisted subset of headers from ThreadContext at the moment of the task creation. This way it would be possible to add stuff on the rest layer in a general way to all requests. Otherwise, each request you would want to "opt in" will have to add a place to "stash" the information you want to expose via task manager api.

[nik9000]

Maybe! If we can get it at the rest layer that'd be cool.

[jrubensteinsp]

This is the single most important feature for our environment. We have users that will run multiple searches in a row, and some are quite large. The ability to tag their searches and cancel specific searches prior to the latest that is still running would be an incredible benefit.

[lusid]

+1! Ability to abort specific prior searches on demand would be huge. I would gladly manage UUIDs on our end and pass them up to just be appended to the task at search time if that means I could do it through a single REST call. The inability to abort ES tasks has been a problem we have had since 0.90.

[daedalus28]

👍 This is critical for our application because we have long-running analytic reports that sometimes are canceled by users, but the es cluster keeps going until it's done - and takes down the cluster in the process because users might immediately queue up different reports now that they've "canceled" the previous one.

[cilerler]

Allowing to assign an ID (not necessary unique one) for a search and be able to cancel it when its needed is crucial for heavy usage scenarios. 💡 Not having it is causing queuing up and leads to search rejections and it literally ties our hand and becomes bottleneck in our operation. 😞 Please make this issue priority 🤗 Thanks in advance

[dshishkov]

It would be really useful to have some sort of control over canceling queries for my use cases too. Thanks for considering it.

[Akrion]

+1