Prioritize FetchPhase over QueryPhase Under High Search Workload #39891
Description
Currently, FetchPhase and QueryPhase will be scheduled on the same threadpool. This can cause issues when dealing with search heavy workloads since FetchPhase can stall behind a slew of QueryPhases. That is problematic since FetchPhase is heavier than QueryPhase (more open search handles for eg), and also means that we have completed the QueryPhase for the specific query, hence the major heavylifting for that query has already been done.
On coordinator nodes, we should complete FetchPhase faster to reduce the probability of throttling kicking in and killing the FetchPhase, hence wasting the resources expended for completing the QueryPhase for this query (#38586). On data nodes, completing FetchPhase earlier should reduce query latencies and should also reduce pressure on memory (#29366).
Another mechanism is to schedule QueryPhase and FetchPhase on different threadpools, but that is discouraged due to the fact that we already have a large number of threadpools today.