Throw errors on search timeouts

[markharwood]

Currently a search timeout will attempt to return partial results along with a timed_out:true flag in the response.
Sadly, today we expect most client applications will pay little attention to this timed_out flag and will present partial results to clients with no indication that the results are incomplete.

This issue proposes we fix this by allowing a new "hard" timeout type which will throw an error to the client application in the event of a timeout. This will require the following:

• A new timeout_type:hard query parameter as part of search requests, as opposed to the default timeout_type:partial(these names are up for debate).
• The default_timeout_type setting would be a new cluster-wide setting allowing admins to define a default policy when clients don't provide any choice on search requests.
• A suitable HTTP response code will be needed for errors - 503 overloaded looks like it may be most appropriate?

[jpountz]

+1 to options for hard timeouts

Naming-wise I think I'd rather like having a timeout and hard_timeout than adding a timeout_type.

[dakrone]

Personally I'd rather stick with a single timeout option, and add a new one for opt-in partial results:

{
  "query": { ... },
  "timeout": "1m",
  "partial_results": true
}

[javanna]

Isn't partial_results too generic though? Elasticsearch returns partial results in other cases but this param enables only partial results coming from timeouts and whether a timeout should fail the whole request or not.

[jimczi]

I agree with @javanna , adding more options will not clarify the meaning of timeout which seems to be the gist of this issue.
I would prefer to change thetimeout option to always be an hard timeout and use another option to represent the time limited query. We already have terminate_after that I find more meaningful than timeout, we could extend the syntax to accept terminate_after=1m.

[dakrone]

I would prefer to change thetimeout option to always be an hard timeout and use another option to represent the time limited query.

Yes, I am in favor of this also

We already have terminate_after that I find more meaningful than timeout, we could extend the syntax to accept terminate_after=1m.

I think that's a great idea, terminate_after use seems much better than introducing a partial_results option.

[markharwood]

If we want to provide for hard+ soft timeouts we need an unambiguous way of stating that.

I'd rather like having a timeout and hard_timeout

Then both could be supplied in a request and we'd need to reject due to ambiguity.

We already have terminate_after

Which responds with a terminated_early response field so I guess we'd effectively have no need for the timed_out field any more if timeout setting changes to throw hard errors?

Removing the timed_out flag then raises the question of BWC for those that are reliant on existing partial results behaviour. They'd need to re-code their apps' response checking logic to look for terminated_early rather than timed_out.
Perhaps the timeout_type approach allows us to preserve the existing API+behaviour and migrate more easily? Early adopters can make use of new hard timeouts and laggards can keep the existing behaviour until we make hard timeouts the default default type (and even then they can change one cluster setting to continue with soft timeouts).

It looks like if might come down to a choice between:

1. Breaking change for better syntax (use terminate_after/terminated_early)
2. Non-breaking change on existing syntax (add a timeout_type)

[markharwood]

Closing in favour of broader flag for tolerating partial results: #27435