[CI] Sporadic failures of testRetentionPolicyChangeDuringRecovery

[DaveCTurner]

I see a couple of recent failures of testRetentionPolicyChangeDuringRecovery:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=opensuse/1797/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+g1gc/5410/console

Also the PR build of #27856:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request/6288/console

Attempting to reproduce as advised yields success for me:

gradle :core:test -Dtests.seed=C56DDBF5A9AE3C38 -Dtests.class=org.elasticsearch.indices.recovery.RecoveryTests -Dtests.method="testRetentionPolicyChangeDuringRecovery" -Dtests.security.manager=true -Dtests.locale=sr-Latn-ME -Dtests.timezone=PRC

Possibly related to work that @dnhatn & @bleskes have been doing in this area? I have no failures of this before 2017-12-16 in my email archive.

[bleskes]

@dnhatn can you research please?

[DaveCTurner]

FWIW repeated runs on my laptop seem fine, but on my build machine it fails quite readily with that REPRODUCE line.

$ uname -a
Linux worker-772973 4.10.0-35-generic #39~16.04.1-Ubuntu SMP Wed Sep 13 09:02:42 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[dnhatn]

Thanks @DaveCTurner. This is reproducible; I will look into it.

[DaveCTurner]

I muted this test in b26cc36 - please could you unmute it as part of the fix?

[dnhatn]

Another instance on 6.x https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=debian/434/console