[CI] RecoveryDuringReplicationTests failures

[davidkyle]

2 recent failures on 6.5 and 6.6 that do not reproduce

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.5+multijob-darwin-compatibility/147/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.6+multijob-darwin-compatibility/37/console

Both failed with the same assertion although the stack trace line numbers are different
6.5

Expected: a value equal to or greater than <50>
     but: <40> was less than <50>
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.index.replication.RecoveryDuringReplicationTests.lambda$testAddNewReplicas$10(RecoveryDuringReplicationTests.java:718)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:848)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:822)
	at org.elasticsearch.index.replication.RecoveryDuringReplicationTests.testAddNewReplicas(RecoveryDuringReplicationTests.java:718)

6.6

Expected: a value equal to or greater than <50>
     but: <48> was less than <50>
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.index.replication.RecoveryDuringReplicationTests.lambda$testAddNewReplicas$10(RecoveryDuringReplicationTests.java:725)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:848)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:822)
	at org.elasticsearch.index.replication.RecoveryDuringReplicationTests.testAddNewReplicas(RecoveryDuringReplicationTests.java:725)

Both report this error

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=409, name=Thread-74, state=RUNNABLE, group=TGRP-RecoveryDuringReplicationTests]
Caused by: java.lang.AssertionError: java.util.concurrent.ExecutionException: [test/uuid][[test][0]] IndexShardClosedException[CurrentState[CLOSED] operation only allowed when not closed]
	at __randomizedtesting.SeedInfo.seed([C20144ADEBC4292B]:0)
	at org.elasticsearch.index.replication.RecoveryDuringReplicationTests.lambda$testAddNewReplicas$9(RecoveryDuringReplicationTests.java:719)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: [test/uuid][[test][0]] IndexShardClosedException[CurrentState[CLOSED] operation only allowed when not closed]
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:265)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:252)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:94)
	at org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationGroup.executeWriteRequest(ESIndexLevelReplicationTestCase.java:245)
	at org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationGroup.index(ESIndexLevelReplicationTestCase.java:228)
	at org.elasticsearch.index.replication.RecoveryDuringReplicationTests.lambda$testAddNewReplicas$9(RecoveryDuringReplicationTests.java:707)
	... 1 more
Caused by: [test/uuid][[test][0]] IndexShardClosedException[CurrentState[CLOSED] operation only allowed when not closed]
	at org.elasticsearch.index.shard.IndexShard.verifyNotClosed(IndexShard.java:1601)
	at org.elasticsearch.index.shard.IndexShard.verifyNotClosed(IndexShard.java:1595)
	at org.elasticsearch.index.shard.IndexShard.getReplicationGroup(IndexShard.java:2006)
	at org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationAction$PrimaryRef.getReplicationGroup(ESIndexLevelReplicationTestCase.java:628)
	at org.elasticsearch.action.support.replication.ReplicationOperation.checkActiveShardCount(ReplicationOperation.java:221)
	at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:91)
	at org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationAction.execute(ESIndexLevelReplicationTestCase.java:570)
	at org.elasticsearch.index.replication.ESIndexLevelReplicationTestCase$ReplicationGroup.executeWriteRequest(ESIndexLevelReplicationTestCase.java:244)
	... 3 more

Reproduce for the 6.5 failures:

./gradlew :server:test \
  -Dtests.seed=E3840CC3FD262F34 \
  -Dtests.class=org.elasticsearch.index.replication.RecoveryDuringReplicationTests \
  -Dtests.method="testAddNewReplicas" \
  -Dtests.security.manager=true \
  -Dtests.locale=es-MX \
  -Dtests.timezone=Africa/Kinshasa \
  -Dcompiler.java=11 \
  -Druntime.java=8

./gradlew :server:test \
  -Dtests.seed=E3840CC3FD262F34 \
  -Dtests.class=org.elasticsearch.index.replication.RecoveryDuringReplicationTests \
  -Dtests.method="testRecoveryAfterPrimaryPromotion" \
  -Dtests.security.manager=true \
  -Dtests.locale=es-MX \
  -Dtests.timezone=Africa/Kinshasa \
  -Dcompiler.java=11 \
  -Druntime.java=8

Reproduce for 6.6:

./gradlew :server:unitTest \
  -Dtests.seed=C20144ADEBC4292B \
  -Dtests.class=org.elasticsearch.index.replication.RecoveryDuringReplicationTests \
  -Dtests.method="testAddNewReplicas" \
  -Dtests.security.manager=true \
  -Dtests.locale=pl-PL \
  -Dtests.timezone=America/Noronha \
  -Dcompiler.java=11 \
  -Druntime.java=8

[elasticmachine]

Pinging @elastic/es-distributed