[CI] DiscoveryWithServiceDisruptionsIT.testAckedIndexing Failure in 5.2

[talevy]

console logs:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.2+multijob-unix-compatibility/os=opensuse/156/console

reproduce:

gradle :core:integTest -Dtests.seed=F8108036B5FA3F4D -Dtests.class=org.elasticsearch.discovery.DiscoveryWithServiceDisruptionsIT -Dtests.method="testAckedIndexing" -Dtests.security.manager=true -Dtests.locale=sv -Dtests.timezone=Etc/GMT+8

stacktrace:

FAILURE 17.7s | DiscoveryWithServiceDisruptionsIT.testAckedIndexing <<< FAILURES!
   > Throwable #1: java.lang.AssertionError:
   > Expected: a value equal to or greater than <5L>
   >      but: <4L> was less than <5L>
   >    at __randomizedtesting.SeedInfo.seed([F8108036B5FA3F4D:72D134C5E987D906]:0)
   >    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >    at org.elasticsearch.discovery.DiscoveryWithServiceDisruptionsIT.lambda$testAckedIndexing$3(DiscoveryWithServiceDisruptionsIT.java:591)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:662)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:636)
   >    at org.elasticsearch.discovery.DiscoveryWithServiceDisruptionsIT.testAckedIndexing(DiscoveryWithServiceDisruptionsIT.java:589)
   >    at java.lang.Thread.run(Thread.java:748)
   >    Suppressed: java.lang.AssertionError:
   > Expected: a value equal to or greater than <5L>
   >      but: <4L> was less than <5L>
   >            at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >            at org.elasticsearch.discovery.DiscoveryWithServiceDisruptionsIT.lambda$testAckedIndexing$3(DiscoveryWithServiceDisruptionsIT.java:591)
   >            at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:650)
   >            ... 39 more
   >    Suppressed: java.lang.AssertionError:

this reproduces for me

[bleskes]

@talevy don't forget to assign people. ;)

@ywelsch do you mind taking a look?

[ywelsch]

Note that this is ES v5.2.
What happened here is that there was a network partition, where a cluster state failed to be published to one of the nodes, and where the node did not notice that it was partitioned from the master, (i.e. the partition was shorter than the fault detection ping interval). This resulted in the cluster on the node being older than the one on the master, tripping an assertion that does not exist like that anymore on the master branch. Maybe fixed by #25320?

Relevant log lines:

1> [2017-07-11T15:28:25,024][INFO ][o.e.d.DiscoveryWithServiceDisruptionsIT] disruption scheme [network disruption (disruption type: network disconnects, disrupted links: bridge partition (super connected node: [node_t0], partition 1: [node_t2] and partition 2: [node_t1]))] added
...
1> [2017-07-11T15:28:25,058][TRACE][o.e.c.s.ClusterService   ] [node_t0] calling [org.elasticsearch.gateway.Gateway@271dd854] with change to version [6]
  1> [2017-07-11T15:28:25,055][DEBUG][o.e.d.z.PublishClusterStateAction] [node_t2] failed to send cluster state to {node_t1}{r4Cc41HqTxCZmgFbEkWRuA}{MvXysgoDRbGP0GViujBZcw}{127.0.0.1}{127.0.0.1:30301}
  1> org.elasticsearch.transport.SendRequestTransportException: [node_t1][127.0.0.1:30301][internal:discovery/zen/publish/send]
  1> 	at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:564) ~[main/:?]
...
1> [2017-07-11T15:28:25,068][INFO ][o.e.t.d.NetworkDisruption] stop disrupting (disruption scheme: network disconnects, disrupted links: bridge partition (super connected node: [node_t0], partition 1: [node_t2] and partition 2: [node_t1]))
...
1> [2017-07-11T15:28:25,085][DEBUG][o.e.d.DiscoveryWithServiceDisruptionsIT] indices [test] are green
...
 > Throwable #1: java.lang.AssertionError: 
   > Expected: a value equal to or greater than <6L>
   >      but: <5L> was less than <6L>
   > 	at __randomizedtesting.SeedInfo.seed([F8108036B5FA3F4D:72D134C5E987D906]:0)
   > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 	at org.elasticsearch.discovery.DiscoveryWithServiceDisruptionsIT.lambda$testAckedIndexing$3(DiscoveryWithServiceDisruptionsIT.java:591)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:662)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:636)
   > 	at org.elasticsearch.discovery.DiscoveryWithServiceDisruptionsIT.testAckedIndexing(DiscoveryWithServiceDisruptionsIT.java:589)
   > 	at java.lang.Thread.run(Thread.java:748)

[bleskes]

@ywelsch thanks for looking. Makes sense to me. I don't know if that PR fixed things as it was response to, I think, more recent refactoring in TcpTransport than 5.2. That said, there is no point in chasing such a race condition in the old code - it's been refactored anyway. I'm closing this.