[ci] RemoveCorruptedShardDataCommandIT.testCorruptIndex and 3 others

[andyb-elastic]

Doesn't repro locally, I'm not sure whether something in the test or the node failing is the root cause here

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=ubuntu&&virtual/97/console

build-unix-97.txt.zip

testCorruptIndex has this in its logging

  1> [2018-12-04T11:42:15,394][INFO ][o.e.i.s.RemoveCorruptedShardDataCommandIT] [testCorruptIndex] after test
ERROR   1.71s J5 | RemoveCorruptedShardDataCommandIT.testCorruptIndex <<< FAILURES!
   > Throwable #1: ElasticsearchException[Shard does not seem to be corrupted at /var/lib/jenkins/workspace/elastic+elasticsearch+master+multijob-unix-compatibility/os/ubuntu&&virtual/server/build/testrun/integTest/J5/temp/org.elasticsearch.index.shard.RemoveCorruptedShardDataCommandIT_7292077B46A25BBF-001/tempDir-002/data/nodes/0/indices/-LSNcjAZQ0uqCXST_25kNA/0]
   >    at org.elasticsearch.index.shard.RemoveCorruptedShardDataCommand.lambda$execute$1(RemoveCorruptedShardDataCommand.java:347)
   >    at org.elasticsearch.index.shard.RemoveCorruptedShardDataCommand.findAndProcessShardPath(RemoveCorruptedShardDataCommand.java:202)
   >    at org.elasticsearch.index.shard.RemoveCorruptedShardDataCommand.execute(RemoveCorruptedShardDataCommand.java:282)
   >    at org.elasticsearch.index.shard.RemoveCorruptedShardDataCommandIT$2.onNodeStopped(RemoveCorruptedShardDataCommandIT.java:192)
   >    at org.elasticsearch.test.InternalTestCluster$NodeAndClient.closeForRestart(InternalTestCluster.java:917)
   >    at org.elasticsearch.test.InternalTestCluster.restartNode(InternalTestCluster.java:1689)
   >    at org.elasticsearch.test.InternalTestCluster.restartNode(InternalTestCluster.java:1649)
   >    at org.elasticsearch.index.shard.RemoveCorruptedShardDataCommandIT.testCorruptIndex(RemoveCorruptedShardDataCommandIT.java:188)
   >    at java.lang.Thread.run(Thread.java:748)

All the tests have this in their logging

java.lang.RuntimeException: already closed
   >    at org.elasticsearch.test.InternalTestCluster$NodeAndClient.client(InternalTestCluster.java:849)
   >    at org.elasticsearch.test.InternalTestCluster.client(InternalTestCluster.java:703)
   >    at org.elasticsearch.test.ESIntegTestCase.client(ESIntegTestCase.java:641)
   >    at org.elasticsearch.test.ESIntegTestCase.client(ESIntegTestCase.java:634)
   >    at org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:553)
   >    at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2191)
   >    at java.lang.Thread.run(Thread.java:748)

./gradlew :server:integTest -Dtests.seed=7292077B46A25BBF -Dtests.class=org.elasticsearch.index.shard.RemoveCorruptedShardDataCommandIT -Dtests.method="testCorruptIndex" -Dtests.security.manager=true -Dtests.locale=ko-KR -Dtests.timezone=Pacific/Tarawa -Dcompiler.java=11 -Druntime.java=8

[elasticmachine]

Pinging @elastic/es-distributed

[andyb-elastic]

Muted on master since this has failed a handful of times over the last month - 01b8f99

[original-brownbear]

I can reproduce this locally by running org.elasticsearch.index.shard.RemoveCorruptedShardDataCommandIT#testCorruptIndex in a loop with constant seed 7292077B46A25BBF (fails about 0.1% of runs). I'll see if I can fix it :)

[original-brownbear]

Fixed in #36208 I think.

[dimitris-athanasiou]

It appears like this also affect 6.7: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+internalClusterTest/135/console.

I don't think we'll be merging zen2 in 6.7, so not sure what to do with this test. Any ideas?