FullClusterRestartIT#testRecovery() fails in CI

[danielmitterdorfer]

build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.2+periodic/803/consoleFull

reproduction line:

./gradlew :qa:full-cluster-restart:v6.1.5-SNAPSHOT#upgradedClusterTestRunner \
  -Dtests.seed=A360128A15AF807 \
  -Dtests.class=org.elasticsearch.upgrades.FullClusterRestartIT \
  -Dtests.method="testRecovery" \
  -Dtests.security.manager=true \
  -Dtests.locale=es-SV \
  -Dtests.timezone=Indian/Kerguelen

failure message:

08:33:13 FAILURE 0.13s | FullClusterRestartIT.testRecovery <<< FAILURES!
08:33:14    > Throwable #1: java.lang.AssertionError: mismatch while checking for translog recovery
08:33:14    > testrecovery 0 existing_store done 0
08:33:14    > testrecovery 0 peer           done 0
08:33:14    > testrecovery 1 existing_store done 0
08:33:14    > testrecovery 1 peer           done 0
08:33:14    > testrecovery 2 existing_store done 0

[elasticmachine]

Pinging @elastic/es-distributed

[bleskes]

@dnhatn do you mind taking a look? Since it's 6.2 - maybe we didn't backport a fix for something?

[dnhatn]

@bleskes I will look at this. Log: testRecovery.txt

[mayya-sharipova]

Another instance of this issue: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+default-distro-bwc-tests/37/console

[dnhatn]

Last instance's log: recovery63.txt

[dnhatn]

Another 6.2 instance: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.2+bwc-tests/415/consoleText. Looking it now.

[jbaiera]

Another 6.2 Instance: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.2+multijob-unix-compatibility/os=centos/555/console

[bleskes]

@dnhatn ping.