Set a default search timeout to clusters #27187
Description
Many client applications overlook the timeout setting in the search API meaning searches run indefinitely.
This issue proposes that a default timeout setting is defined for search.default_search_timeout.
What will be a good choice of default setting is an interesting question - many applications that exist today will expect sub-second search responses but several may be happy with searches that can take up to a minute.
Assuming we pick "1 minute" as the default timeout default then we run the risk of returning partial results to many existing applications that are content with >1 minute response times and have no code to check for timed_out responses. They will be lying to their users about stats at this point. Better then, that we throw an error if searches timeout by default using the new setting proposed in #27186
It would be better to loudly complain about excessive resource usage on our cluster than silently return incorrect/partial results.