[CI] ReindexFailureTests testAbortOnVersionConflict, testFailuresCauseAbortDefault, testResponseOnSearchFailure

[andyb-elastic]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=centos/479/console

build-479-ReindexFailureTests.txt

ERROR   51.4s J3 | ReindexFailureTests.testResponseOnSearchFailure <<< FAILURES!
   > Throwable #1: java.lang.AssertionError:
   > Expected: an empty iterable
   > dices:data/write/bulk[s]]]; nested: IndexNotFoundException[no such index];>,<RemoteTransportException[[node_sd3][127.0.0.1:35162][indices:data/write/bulk[s]]]; nested: IndexNotFoundException[no such index];>,<RemoteTransportException[[node_sd3][127.0.0.1:35162][indices:data/write/bulk[s]]]; nested: IndexNotFoundException[no such index];>,<RemoteTransportException[[node_sd3][127.0.0.1:35162][indices:data/write/bulk[s]]]; nested: IndexNotFoundException[no such index];>,<RemoteTransportException[[node_sd3][127.0.0.1:35162][indices:data/write/bulk[s]]]; nested: IndexNotFoundException[no such index];>,<RemoteTransportException[[node_sd3][127.0.0.1:35162][indices:data/write/bulk[s]]]; nested: IndexNotFoundException[no such index];>,<RemoteTransportException[[node_sd3][127.0.0.1:35162][indices:data/write/bulk[s]]]; nested: IndexNotFoundException[no such index];>,<RemoteTransportException[[node_sd3][127.0.0.1:35162][indices:data/write/bulk[s]]]; nested: IndexNotFoundException[no such index];>,<RemoteTransportException[[node_sd1][127.0.0.1:43434][indices:data/write/index]]; nested: RemoteTransportException[[node_sd3][127.0.0.1:35162][indices:data/write/bulk[s]]]; nested: IndexNotFoundException[no such index];>,<RemoteTransportException[[node_sd2][127.0.0.1:35287][indices:data/write/bulk[s]]]; nested: IndexNotFoundException[no such index];>,<RemoteTransportException[[node_sd1][127.0.0.1:43434][indices:data/write/index]]; nested: RemoteTransportException[[node_sd3][127.0.0.1:35162][indices:data/write/bulk[s]]]; nested: IndexNotFoundException[no such index];>,<RemoteTransportException[[node_sd2][127.0.0.1:35287][indices:data/write/bulk[s]]]; nested: IndexNotFoundException[no such index];>,<RemoteTransportException[[node_sd2][127.0.0.1:35287][indices:data/write/bulk[s]]]; nested: IndexNotFoundException[no such index];>,<RemoteTransportException[[node_sd2][127.0.0.1:35287][indices:data/write/bulk[s]]]; nested: IndexNotFoundException[no such index];>,<RemoteTransportException[[node_sd2][127.0.0.1:35287][indices:data/write/bulk[s]]]; nested: IndexNotFoundException[no such index];>,<RemoteTransportException[[node_sd2][127.0.0.1:35287][indices:data/write/bulk[s]]]; nested: IndexNotFoundException[no such index];>,<RemoteTransportException[[node_sd2][127.0.0.1:35287][indices:data/write/bulk[s]]]; nested: IndexNotFoundException[no such index];>,<RemoteTransportException[[node_sd1][127.0.0.1:43434][indices:data/write/index]]; nested: RemoteTransportException[[node_sd2][127.0.0.1:35287][indices:data/write/bulk[s]]]; nested: IndexNotFoundException[no such index];>,<RemoteTransportException[[node_sd2][127.0.0.1:35287][indices:data/write/bulk[s]]]; nested: IndexNotFoundException[no such index];>,<RemoteTransportException[[node_sd2][127.0.0.1:35287][indices:data/write/bulk[s]]]; nested: IndexNotFoundException[no such index];>,<[source/sj2mv1B6SP6OUxseTSjeOQ] IndexNotFoundException[no such index]>,<[source/sj2mv1B6SP6OUxseTSjeOQ] IndexNotFoundException[no such index]>,<[source/sj2mv1B6SP6OUxseTSjeOQ] IndexNotFoundException[no such index]>,<[source/sj2mv1B6SP6OUxseTSjeOQ] IndexNotFoundException[no such index]>,<RemoteTransportException[[node_sd1][127.0.0.1:43434][indices:data/write/index]]; nested: IndexNotFoundException[no such index];>,<[source/sj2mv1B6SP6OUxseTSjeOQ] IndexNotFoundException[no such index]>,<[source/sj2mv1B6SP6OUxseTSjeOQ] IndexNotFoundException[no such index]>,<[source/sj2mv1B6SP6OUxseTSjeOQ] IndexNotFoundException[no such index]>,<[source/sj2mv1B6SP6OUxseTSjeOQ] IndexNotFoundException[no such index]>,<[source/sj2mv1B6SP6OUxseTSjeOQ] IndexNotFoundException[no such index]>,<[source/sj2mv1B6SP6OUxseTSjeOQ] IndexNotFoundException[no such index]>,<[source/sj2mv1B6SP6OUxseTSjeOQ] IndexNotFoundException[no such index]>,<[source/sj2mv1B6SP6OUxseTSjeOQ] IndexNotFoundException[no such index]>,<[source/sj2mv1B6SP6OUxseTSjeOQ] IndexNotFoundException[no such index]>,<[source/sj2mv1B6SP6OUxseTSjeOQ] IndexNotFounption[no such index]>,<[source/sj2mv1B6SP6OUxseTSjeOQ] IndexNotFoundException[no such index]>,<[source/sj2mv1B6SP6OUxseTSjeOQ] IndexNotFoundException[no such index]>,<[source/sj2mv1B6SP6OUxseTSjeOQ] IndexNotFoundException[no such index]>]
   >    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >    at org.elasticsearch.test.ESIntegTestCase.indexRandom(ESIntegTestCase.java:1451)
   >    at org.elasticsearch.test.ESIntegTestCase.indexRandom(ESIntegTestCase.java:1369)
   >    at org.elasticsearch.test.ESIntegTestCase.indexRandom(ESIntegTestCase.java:1353)
   >    at org.elasticsearch.index.reindex.ReindexFailureTests.indexDocs(ReindexFailureTests.java:128)
   >    at org.elasticsearch.index.reindex.ReindexFailureTests.testResponseOnSearchFailure(ReindexFailureTests.java:100)
   >    at java.lang.Thread.run(Thread.java:748)Throwable #2: java.lang.AssertionError: There are still [1] in-flight contexts. The first one's creation site is listed as the cause of this exception.
   >    at org.elasticsearch.search.MockSearchService.assertNoInFlightContext(MockSearchService.java:51)
   >    at org.elasticsearch.test.ESTestCase.lambda$ensureAllSearchContextsReleased$4(ESTestCase.java:421)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:731)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:705)
   >    at org.elasticsearch.test.ESTestCase.ensureAllSearchContextsReleased(ESTestCase.java:421)
   >    at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2072)
   >    at java.lang.Thread.run(Thread.java:748)
   >    Suppressed: java.lang.AssertionError: There are still [1] in-flight contexts. The first one's creation site is listed as the cause of this exception.
   >            at org.elasticsearch.search.MockSearchService.assertNoInFlightContext(MockSearchService.java:51)
   >            at org.elasticsearch.test.ESTestCase.lambda$ensureAllSearchContextsReleased$4(ESTestCase.java:421)
   >            at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
   >            ... 37 more

gradle :modules:reindex:test \
  -Dtests.seed=401B0C5B3206920F \
  -Dtests.class=org.elasticsearch.index.reindex.ReindexFailureTests \
  -Dtests.method="testAbortOnVersionConflict" \
  -Dtests.security.manager=true \
  -Dtests.locale=sr-Latn \
  -Dtests.timezone=America/Cuiaba

[nik9000]

Ok. I can't reproduce this one but I have a theory and I'll push a clean up that might prevent this in the future. Might.

[nik9000]

I pushed eded5bc to try and prevent this from happening again but it might not work. I wasn't able to reproduce locally so it is kind of a shot in the dark.

[dakrone]

I searched through my mail and wasn't able to find another failure from this after the push Nik made, so I am closing this for now.

[colings86]

This seems to have occurred again in http://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/2202