Improve robustness of monitoring APIs

[imotov]

We observed some cases (#50241 for example) where a data node responding slowly can cause accumulation of ResponseContexts for indices:monitor/recovery[n], indices:monitor/stats[n], cluster:monitor/stats[n] and cluster:monitor/xpack/ml/job/stats/get[n] which correspond to _xpack/usage and _nodes/stats calls.

We would like to improve robustness of stats and usage call in case of a slowly responding data nodes by

1. introducing timeout on stats and usage APIs and/or
2. making stats and usage APIs tasks cancellable and cancel them if the REST client disconnects

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Monitoring)

[imotov]

Just an additional data point. It looks like the metricbeat's elaticsearch module has 10s timeout. So if we don't deliver stats in 10 seconds, it doesn't care anymore.

[DaveCTurner]

Option 2 means that clients can safely implement their own end-to-end timeouts, which means we don't need option 1; conversely if we do option 1 then that still requires clients to implement their own timeouts to protect against requests getting lost on the way to Elasticsearch. I therefore prefer option 2 alone.

Relates #60188 which is pretty much the same issue but for the transport-like client that the internal monitoring system uses, which wouldn't be solved by cancelling tasks on a client disconnection since the internal client doesn't disconnect like that.

Also relates #51992.

[dhwanilpatel]

Hello Elastic team,

Is team working on this or expecting any community contribution?
If team is already working, Can you please provide any timeline for it? It will be very helpful. Thanks!

[DaveCTurner]

Copying here the info that @Bukhtawar intends to work on cancellation of the various stats APIs, see #66992 (comment).

@Bukhtawar you might be interested in #67413 which implements the right sort of behaviour on a different API.