Do not attempt to re-establish connections when applying a cluster state update

[DaveCTurner]

Today, when a new cluster state is applied, we attempt to connect to all nodes in the new cluster state:

elasticsearch/server/src/main/java/org/elasticsearch/cluster/service/ClusterApplierService.java

Line 467 in 5904d93

nodeConnectionsService.connectToNodes(newClusterState.nodes());

This set of nodes may include some nodes that were previously connected but which have failed, but whose failure is to be processed in a later cluster state update. Attempts to reconnect to these nodes will also fail, but may do so very slowly if they are unresponsive. On the other hand, the NodesConnectionService takes responsibility for periodically re-establishing connections that have dropped.

This means that there's no real need to attempt to re-establish these connections while applying a cluster state update, and #28920 is an example of a situation where it's undesirable to do so. Therefore it seems sensible to skip these nodes.

NB it's important that the nodes in the cluster state align with the nodes known to the NodeConnectionsService, so they can't simply be omitted in the call to connectToNodes() above.

[elasticmachine]

Pinging @elastic/es-distributed