Async search #49091
Description
This is a meta issue to monitor progress around adding support for asynchronous search.
The main goal is to launch a search request and, instead of waiting for its final response, let it run asynchronously while being able to monitor its progress. The progress of an async search includes how many operations have been performed compared to the total number of operations required for completion, as well as intermediate results that may be available before completion.
The high-level steps required for this feature are listed as follows, together with their corresponding licensing:
- add ability to control the order in which shards are queried by the search API (OSS) Pre-sort shards based on the max/min value of the primary sort field #49092
- add ability to monitor progress of a search request through a listener that gets notified whenever new shard results are available (OSS) Add a listener to track the progress of a search request locally #49471
- introduce new endpoints to submit, get and delete an async search (Basic) Add new x-pack endpoints to track the progress of a search asynchronously #49931
- Add tests for the AsyncSearchMaintenanceService
- Add documentation for the APIs
- Add support for the HLRC.
The resiliency of async search is besides the scope of the initial implementation. If the coordinating node goes down while an async search is running, the most recent intermediate results should be made available for retrieval, but the async search is aborted and never retried, hence it needs to be restarted manually.