Cancel task (and descendants) if its originating transport request times out

[DaveCTurner]

If a sender sets a timeout on a transport request and does not receive a response in time then today we make no attempt to inform the receiver that we no longer care about its response. This is particularly bad for stats requests that may be timing out on one broken node, but still continue to pile up there since that node has no way to know that these requests are now irrelevant and should not be processed.

A couple of possible solutions spring to mind:

• When the sender times out it could sends a task cancellation request to the receiver.

• The sender could indicate the timeout to the receiver, which could then implement its own local timeout-and-cancel behaviour.

Relates #60188, #52616, #51992.

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[Bukhtawar]

Thanks @DaveCTurner, Is it a prioritised item already, if not I'd be interested to work on this

[DaveCTurner]

We're still contemplating how/whether to proceed on this. We may take no action since this only really applies to internally-generated requests which are mostly fairly well-behaved already. @Bukhtawar perhaps you'd be interested in resolving #55550 instead? Supporting cancellation on stats APIs would address the concerns you raised in #52616 for instance.

[Bukhtawar]

Sure @DaveCTurner I'll take up support for cancellation on stats API. I'll get back on this