Give higher priorities to fetch tasks than query tasks?

[jpountz]

Under high search load, fetch tasks may have to wait quite some time in the queue before being processed. I was wondering that we might want to give them a higher priority than query tasks in order to minimize the amount of time during which we have to keep resources open at the shard level, such as the index reader, and the amount of time during which the coordinating node needs to keep shard responses in memory. However I wouldn't expect it to improve latency or throughput.

Note that while this might look related to #14224, this issue doesn't try to give some searches a higher priority than other searches, but to different tasks that are required to run a single search.

[elasticmachine]

Pinging @elastic/es-search-aggs

[jpountz]

We discussed it further in FixitFriday and within the search team. Another option that was mentioned was to split tasks into two threadpools but we decided against it because we want to keep the number of threads contained, and decreasing the number of threads that query tasks can run in doesn't sound good either since some workloads consist only of query tasks.

So next steps would be the following:

• add metrics regarding the amount of time during which we keep readers open after they have been replaced by a new reader (refresh)
• try to introduce priorities for query vs fetch tasks, either by using some sort of priority queue or by wrapping 2 queues and taking from the one that contains fetch tasks first
• benchmark with Rally

[atris]

Folks, I would like to take a crack at this. Could someone please assign it to me?

[WhiteBlackX6]

Hello, Can someone assign this to me? I would really like to contribute to this project.

[jpountz]

For the record, I don't think this issue is a good first issue. You can search for issues with the "good first issue" label if you are looking for something to work on.

[atris]

@jpountz Could you share your thoughts on what kind of Rally workloads can trigger this? I have a half baked patch that I want to run through rally first and see how things are looking before moving ahead with the patch. I tried running http_logs, but it doesnt seem search heavy enough to warrant this behaviour.

[jpountz]

Good question. This change would only matter if the search queue is not empty so we'd need to generate significant search load.