[CI] SearchWhileCreatingIndexIT#testNoReplicas fails assertion, hits count is 0 but 1 was expected

[polyfractal]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+g1gc/3069/console

Looks like the primary shard fails to return the expected document during the test, but I could not get this to reproduce locally.

@jasontedor looks like you've fiddled with this test in the past, is this up your alley? Or know who to delegate to?

gradle :core:integTest -Dtests.seed=916B2EE6CB28B85F -Dtests.class=org.elasticsearch.search.basic.SearchWhileCreatingIndexIT -Dtests.method="testNoReplicas" -Dtests.security.manager=true -Dtests.jvm.argline="-XX:-UseConcMarkSweepGC -XX:+UseG1GC" -Dtests.locale=en-SG -Dtests.timezone=America/Inuvik

java.lang.AssertionError: Count is 0 but 1 was expected.  Total shards: 6 Successful shards: 5 & 0 shard failures:
	at __randomizedtesting.SeedInfo.seed([916B2EE6CB28B85F:7F1DCA6C9136A733]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertHitCount(ElasticsearchAssertions.java:262)
	at org.elasticsearch.search.basic.SearchWhileCreatingIndexIT.searchWhileCreatingIndex(SearchWhileCreatingIndexIT.java:80)
	at org.elasticsearch.search.basic.SearchWhileCreatingIndexIT.testNoReplicas(SearchWhileCreatingIndexIT.java:44)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)

[jaymode]

This failed today but in a different method https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.4+multijob-unix-compatibility/os=amazon/171/console

The failure looks very similar to this one.

[dimitris-athanasiou]

Another failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=fedora/1431/consoleFull

[ywelsch]

Another failure, this time due to another reason (I suspect #26791):

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+periodic/4580/consoleFull

08:43:56 ERROR   0.56s J1 | SearchWhileCreatingIndexIT.testNoReplicas <<< FAILURES!
08:43:56    > Throwable #1: java.lang.IllegalArgumentException: no data nodes with criteria [tinb1WwBTCim7A7_PmvSow] found for shard: [test][6]
08:43:56    > 	at __randomizedtesting.SeedInfo.seed([4A45B266862A58FD:A43356ECDC344791]:0)
08:43:56    > 	at org.elasticsearch.cluster.routing.IndexShardRoutingTable.onlyNodeSelectorActiveInitializingShardsIt(IndexShardRoutingTable.java:489)
08:43:56    > 	at org.elasticsearch.cluster.routing.OperationRouting.preferenceActiveShardIterator(OperationRouting.java:205)
08:43:56    > 	at org.elasticsearch.cluster.routing.OperationRouting.searchShards(OperationRouting.java:102)
08:43:56    > 	at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:288)
08:43:56    > 	at org.elasticsearch.action.search.TransportSearchAction.lambda$doExecute$4(TransportSearchAction.java:193)
08:43:56    > 	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59)
08:43:56    > 	at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:113)
08:43:56    > 	at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:86)
08:43:56    > 	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:213)
08:43:56    > 	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:68)
08:43:56    > 	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
08:43:56    > 	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
08:43:56    > 	at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:79)
08:43:56    > 	at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:69)
08:43:56    > 	at org.elasticsearch.transport.AssertingTransportInterceptor$1.messageReceived(AssertingTransportInterceptor.java:76)
08:43:56    > 	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67)
08:43:56    > 	at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1526)
08:43:56    > 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
08:43:56    > 	at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:135)
08:43:56    > 	at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1483)
08:43:56    > 	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1354)
08:43:56    > 	at org.elasticsearch.transport.nio.TcpReadHandler.handleMessage(TcpReadHandler.java:38)
08:43:56    > 	at org.elasticsearch.transport.nio.channel.TcpReadContext.read(TcpReadContext.java:83)
08:43:56    > 	at org.elasticsearch.transport.nio.SocketEventHandler.handleRead(SocketEventHandler.java:95)
08:43:56    > 	at org.elasticsearch.transport.nio.SocketSelector.handleRead(SocketSelector.java:145)
08:43:56    > 	at org.elasticsearch.transport.nio.SocketSelector.processKey(SocketSelector.java:67)
08:43:56    > 	at org.elasticsearch.transport.nio.ESSelector.singleLoop(ESSelector.java:115)
08:43:56    > 	at org.elasticsearch.transport.nio.ESSelector.runLoop(ESSelector.java:80)
08:43:56    > 	at java.lang.Thread.run(Thread.java:748)

The original failure in this ticket was fixed by 4b813ad

[ywelsch]

Other tests are also failing in this class for the same reason:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/588/consoleFull

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/587/console

[dnhatn]

The new failures were fixed by 8474269

[dnhatn]

I think this should be closed by 8474269