[CI] Reproducible Failure in CoordinatorTests.testUnresponsiveFollowerDetectedEventually

[original-brownbear]

./gradlew ':server:test' --tests "org.elasticsearch.cluster.coordination.CoordinatorTests.testUnresponsiveFollowerDetectedEventually" -Dtests.seed=B0D0BD93A4DAC474 -Dtests.security.manager=true -Dtests.locale=mt -Dtests.timezone=Etc/GMT-4 -Druntime.java=11

failed in https://gradle-enterprise.elastic.co/s/gghxzf4mny3wq/tests/:server:test/org.elasticsearch.cluster.coordination.CoordinatorTests/testUnresponsiveFollowerDetectedEventually#1

with

  1> [2020-08-12T18:18:40,656][INFO ][o.e.c.c.CoordinatorTests ] [testUnresponsiveFollowerDetectedEventually] after test
  2> REPRODUCE WITH: ./gradlew ':server:test' --tests "org.elasticsearch.cluster.coordination.CoordinatorTests.testUnresponsiveFollowerDetectedEventually" -Dtests.seed=B0D0BD93A4DAC474 -Dtests.security.manager=true -Dtests.locale=mt -Dtests.timezone=Etc/GMT-4 -Druntime.java=11
  2> java.lang.AssertionError: node1 is a follower of node4
    Expected: is <FOLLOWER>
         but: was <CANDIDATE>
        at __randomizedtesting.SeedInfo.seed([B0D0BD93A4DAC474:D9BF60193F135F6A]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.elasticsearch.cluster.coordination.AbstractCoordinatorTestCase$Cluster.stabilise(AbstractCoordinatorTestCase.java:535)
        at org.elasticsearch.cluster.coordination.AbstractCoordinatorTestCase$Cluster.stabilise(AbstractCoordinatorTestCase.java:495)
        at org.elasticsearch.cluster.coordination.CoordinatorTests.testUnresponsiveFollowerDetectedEventually(CoordinatorTests.java:681)
  2> NOTE: leaving temporary files on disk at: /Users/armin/src/elasticsearch/server/build/testrun/test/temp/org.elasticsearch.cluster.coordination.CoordinatorTests_B0D0BD93A4DAC474-001
  2> NOTE: test params are: codec=Asserting(Lucene86), sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=mt, timezone=Etc/GMT-4
  2> NOTE: Mac OS X 10.15.6 x86_64/Oracle Corporation 11 (64-bit)/cpus=12,threads=1,free=484459008,total=536870912
  2> NOTE: All tests run in this JVM: [CoordinatorTests]

This only failed once ever and for the first time today it seems.
Also, this is reproducible locally, I'll take a look.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Cluster Coordination)

[original-brownbear]

This was caused by #60873. Without a join timeout on the join request a black-holed connection that quietly drops requests during cluster setup breaks the join functionality. This is a problem with the test simulating an unrealistic transport scenario => on it for a fix