IndexRecoveryIT testTransientErrorsDuringRecoveryAreRetried timeout

[davidkyle]

Build scan:
https://gradle-enterprise.elastic.co/s/44qx5vkr5yyta

Repro line:
./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.indices.recovery.IndexRecoveryIT.testTransientErrorsDuringRecoveryAreRetried"
-Dtests.seed=94212EE7FF83A4A
-Dtests.security.manager=true
-Dtests.locale=es-PE
-Dtests.timezone=Cuba
-Druntime.java=11

Reproduces locally?:
No

Applicable branches:
master

Failure history:
At first I put this down to a slow test machine but build stats shows that something significant happened on May 27th

https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-30d,mode:quick,to:now))&_a=(columns:!(build.branch),index:b646ed00-7efc-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:testTransientErrorsDuringRecoveryAreRetried),sort:!(process.time-start,desc))

Failure excerpt:

java.lang.AssertionError: timed out waiting for green state
	at __randomizedtesting.SeedInfo.seed([94212EE7FF83A4A:60A8FEB1B9FED2DE]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.elasticsearch.test.ESIntegTestCase.ensureColor(ESIntegTestCase.java:892)
	at org.elasticsearch.test.ESIntegTestCase.ensureGreen(ESIntegTestCase.java:845)
	at org.elasticsearch.test.ESIntegTestCase.ensureGreen(ESIntegTestCase.java:834)
	at org.elasticsearch.indices.recovery.IndexRecoveryIT.testTransientErrorsDuringRecoveryAreRetried(IndexRecoveryIT.java:823)

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Recovery)

[benwtrent]

Another build failure:

Build: https://gradle-enterprise.elastic.co/s/33lacsqne553k

Reproduce:

./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.indices.recovery.IndexRecoveryIT.testTransientErrorsDuringRecoveryAreRetried" -Dtests.seed=8E1179DC9951D782 -Dtests.security.manager=true -Dtests.locale=nl-BE -Dtests.timezone=America/Guadeloupe -Druntime.java=11

Trace:

org.elasticsearch.indices.recovery.IndexRecoveryIT > testTransientErrorsDuringRecoveryAreRetried FAILED
    java.lang.AssertionError: timed out waiting for green state
        at __randomizedtesting.SeedInfo.seed([8E1179DC9951D782:E7FB95835F573F16]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.elasticsearch.test.ESIntegTestCase.ensureColor(ESIntegTestCase.java:892)
        at org.elasticsearch.test.ESIntegTestCase.ensureGreen(ESIntegTestCase.java:845)
        at org.elasticsearch.test.ESIntegTestCase.ensureGreen(ESIntegTestCase.java:834)
        at org.elasticsearch.indices.recovery.IndexRecoveryIT.testTransientErrorsDuringRecoveryAreRetried(IndexRecoveryIT.java:823)