[CI] MinimumMasterNodesIT.testMultipleNodesShutdownNonMasterNodes sporadically fails

[imotov]

The testMultipleNodesShutdownNonMasterNodes test in org.elasticsearch.cluster.MinimumMasterNodesIT suite started to fail regularly. It failed 6 times over the weekend and once today. I cannot reproduce it locally.

The failure is

  1> [2016-12-12T09:11:56,643][INFO ][o.e.c.MinimumMasterNodesIT] [testMultipleNodesShutdownNonMasterNodes]: after test
ERROR   4.89s J1 | MinimumMasterNodesIT.testMultipleNodesShutdownNonMasterNodes <<< FAILURES!
   > Throwable #1: ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];]
   >    at __randomizedtesting.SeedInfo.seed([4489C6FFC6B9860E:F68A1C476DB0C1A]:0)
   >    at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:161)
   >    at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:147)
   >    at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:92)
   >    at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:52)
   >    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170)
   >    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
   >    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84)
   >    at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:75)
   >    at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:64)
   >    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:400)
   >    at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:77)
   >    at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:51)
   >    at org.elasticsearch.cluster.MinimumMasterNodesIT.testMultipleNodesShutdownNonMasterNodes(MinimumMasterNodesIT.java:264)
   >    at java.lang.Thread.run(Thread.java:745)

[javanna]

Not sure this is happens sporadically, I saw this a couple of times today out of probably 10 runs.

[ywelsch]

This test failure is triggered by PR #22049 which simplifies the cluster state update logic to only change the list of disco-nodes in the cluster state when said cluster state is going to be published. This means that a node cannot just locally change its cluster state and state changes always have to go through an active master. The test failure here highlights a few issues that are not only bound to happen with the above change (but which makes them more frequent):

• Validating full transport connections: Pinging reuses an existing connection when it's available. It does not re-check however whether the node on the other side is who it thinks it is when reconnecting. This is an issue if the cluster state has disco-nodes in it that are dead and when there are new live nodes which have the same transport address but different node id as the dead nodes and the pinging just reconnects to the different node.
• Master election gets confused by dead nodes: The reason is that master election proceeds in two steps: 1) Pinging to determine a master and collecting joins 2) Publishing a cluster state which sets the new master and which needs "approval" from the master-eligible nodes through the process of committing. These two steps can disagree, however, when we elect a master based on nodes with same transport address (but different node id) as previous nodes which are now dead. I've opened a PR (Prefer joining node with conflicting transport address when becoming master #22134) so that the live nodes take precedence over the dead nodes when completing the master-election process, so that the cluster state electing the master is published and committed by the right nodes.
• Async shard fetching is brittle in presence of the issues above. The test actually fails because it ends up with unassigned shards. The reason for this is that the update logic for the cache used by the async shard fetching does not properly handle the scenario where target nodes of a shard fetch do not correspond to the nodes that the master (who is issuing the fetch) believes them to be. When the master starts fetching it adds a NodeEntry to the cache keyed by the node id. When a response is then received, it takes the node id that is contained in the response object to resolve the cache entry. In a situation where a node has a different node id than what the master believes it to have but the same transport address, the response will just be ignored and also prevent future fetching on this node id as the master thinks that fetching is still going on for that node. As such shard fetching will never complete and shards cannot be allocated as they're waiting on the fetching to complete.