Send ban parent per outstanding child connection

[dnhatn]

Few things that prevent us from supporting cross-clusters tasks cancellation:

1. A child task gets canceled if a node with the parent task leaves the cluster. This doesn't hold for a cross-clusters task as the parent task does not belong to the remote cluster. This issue was resolved in Cancel task and descendants on channel disconnects #56620, where we cancel child tasks on disconnect instead.

2. Ban action isn't registered on proxy nodes

3. Banned parent markers are removed when a node with the parent task leaves the cluster. This issue is similar to the first issue.

This commit tries to address the third issue by periodically (every 5 minutes) sending heartbeats for banned parent markers and removing markers that haven't received heartbeats for some time (30 minutes).

This commit tries to address the third issue by sending a ban for each outstanding child connection and removing it when all associated connections disconnect.

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[dnhatn]

@DaveCTurner I looked into your proposal to remove a ban once all associated child connections are disconnected. However, it isn't watertight as we can remove a ban before an in-flight child task start. Any suggestions are welcomed.

[DaveCTurner]

However, it isn't watertight as we can remove a ban before an in-flight child task start. Any suggestions are welcomed.

Hmm wouldn't that mean the child task is going to respond on a connection that was closed before the task even started? If so we shouldn't even start the task.

[dnhatn]

I should have been more clear. We do not prevent a child task from starting in this scenario:

1. The parent task sends message m1 to start a child task t1 on node-1 using connection c1
2. The parent task also sends message m2 to start a child task t2 on node-1 using connection c2
3. Message m1 arrives, and node-1 starts child task t1
4. The parent task is canceled
5. We send a ban message b1 to node-1 (to cancel or prevent t1/t2 from starting)
6. If the connection c1 is disconnected and we remove the ban b1, then node-1 will start child task t2 when the message m2 arrives

[DaveCTurner]

I see. Could we indicate in the ban message that there are already in-flight tasks on connections c1 and c2, so it doesn't get removed while c2 is still open?

[DaveCTurner]

On reflection that sounds hard, we have no way to refer to a specific connection that the other end can understand, except of course sending the ban over every relevant connection.

[dnhatn]

Another issue is proxy connections. A proxy connection consists of two connections: the source node to the proxy node and the proxy node to the target node. While the former will never change, the latter can change anytime. I think we can't use the connection info to remove ban markers.

[DaveCTurner]

Right, that was what I meant earlier about having the proxy also proxy a disconnect event. We can't describe a specific set of connections to a remote cluster, but we can say "the connection on which this message was received".

[dnhatn]

@DaveCTurner I implemented a prototype that does not require heartbeats for the ban markers. In the prototype, we send a ban for each outstanding child connection and remove bans when all associated connections of that ban disconnect.

I've updated this PR to send bans per child connection and will open a follow-up to associate connections to bans. This PR is ready for review again. Can you please take another look? Thank you!

[dnhatn]

I also included (but reverted from this PR) the commit 143620d that tracks bans for each channel and removes them when their associated channels are closed.

[DaveCTurner]

Hmm, I think this still isn't quite watertight, because a Connection is a bundle of channels so messages to a single Connection are not fully ordered. I think it's possible that we could send a task T on one channel that takes a very long time to arrive, then on a different channel send the ban and receive a response, then clear the ban and receive a response, and only then does task T arrive at the target.

Edit: I see that I caused this confusion: I said "connection" in the earlier discussion. I meant TCP connection, i.e. channel.

[dnhatn]

@DaveCTurner Thanks for looking. That's a good point. I think we can make it tighter by sending a ban for every pair of Connection and TransportRequestOptions.Type, which is used to pick a TCP channel. WDYT?

[DaveCTurner]

Ok thanks for confirming, this LGTM then. Neatly done, I like this change 😄

[dnhatn]

@DaveCTurner Thanks so much for reviewing and suggesting this direction :)

[dnhatn]

@elasticmachine update branch