Avoid timeout-and-retry in CoordinationDiagnosticsService and friends

[DaveCTurner]

Both o.e.c.c.CoordinationDiagnosticsService#sendTransportRequest and o.e.c.c.MasterHistoryService#refreshRemoteMasterHistory use transport-layer timeouts to fail a request if it takes too long. A timeout in this area makes sense, because we want to know if the remote node isn't responding promptly and we don't want to see arbitrarily-stale responses, but a transport-layer timeout is problematic because it means we lose track of the in-flight request so we might just keep retrying and piling up work on the handling node, making things worse and worse.

I think we should avoid using transport-layer timeouts here and ensure that we only ever have one request in flight at once. We can still use something like o.e.a.s.SubscribableListener#addTimeout to fail as expected and ensure we don't see arbitrarily-stale responses, but we should still wait for the transport response before sending another request.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[masseyke]

Has the thinking on best practices changed? I was thinking we'd talked about this before and found #85941 (comment). :) I'll take a look at SubscribableListener.

[DaveCTurner]

Yes sorry, I didn't realise these things might be as expensive as they seem to be today.

[masseyke]

Let me make sure I'm understanding you correctly -- we want to notify the listener of failure after 10 seconds, but to leave the transport action running (which I think is what SubscribableListener will do). And we don't want to use the transport action's results if they come back after more than 10 seconds. But we also don't want want to call the transport action again while the stale one is still running. If that's all true, then a couple of questions:

• Is the problem that TransportRequestOptions.timeout() does not actually result in the transport action being canceled on the remote machine? I had thought that it did (but I hadn't ever looked into it).
• What would we want to do if another request came in and the abandoned transport action was still running? Immediately call onFailure()? Wait to see if it finishes and we can call it again and get a fresh result all within 10 seconds?