[CI] SearchStatsIT & IndexingIT failures

[davidkyle]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/1090/console

This is a reproducible failure that errors with

  2> java.lang.AssertionError
  2>    at __randomizedtesting.SeedInfo.seed([33185699574722A3]:0)
  2>    at org.elasticsearch.action.search.SearchPhaseController$TopDocsStats.getTotalHits(SearchPhaseController.java:763)
  2>    at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:510)
  2>    at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:421)
  2>    at org.elasticsearch.action.search.SearchPhaseController.reducedScrollQueryPhase(SearchPhaseController.java:412)
  2>    at org.elasticsearch.action.search.SearchScrollQueryThenFetchAsyncAction$1.run(SearchScrollQueryThenFetchAsyncAction.java:71)
  2>    at org.elasticsearch.action.search.SearchScrollAsyncAction$1.innerOnResponse(SearchScrollAsyncAction.java:189)
  2>    at org.elasticsearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:45)

./gradlew :server:integTest \
  -Dtests.seed=33185699574722A3 \
  -Dtests.class=org.elasticsearch.search.stats.SearchStatsIT \
  -Dtests.method="testOpenContexts" \
  -Dtests.security.manager=true \
  -Dtests.locale=ga-IE \
  -Dtests.timezone=Africa/Windhoek \
  -Dcompiler.java=11 \
  -Druntime.java=8

[elasticmachine]

Pinging @elastic/es-search

[davidkyle]

Muted on master in 7cc749d

[davidkyle]

FieldLevelSecurityTests failed with the same assertion.

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=java11,nodes=virtual&&linux/165/console

Does not reproduce

./gradlew :x-pack:plugin:security:unitTest \
  -Dtests.seed=6624CB3E3B1A0E76 \
  -Dtests.class=org.elasticsearch.integration.FieldLevelSecurityTests \
  -Dtests.method="testScroll" \
  -Dtests.security.manager=true \
  -Dtests.locale=sr-Latn-ME \
  -Dtests.timezone=Africa/Malabo \
  -Dcompiler.java=12 \
  -Druntime.java=11

Caused by: java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([6624CB3E3B1A0E76]:0)
	at org.elasticsearch.action.search.SearchPhaseController$TopDocsStats.getTotalHits(SearchPhaseController.java:763)
	at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:510)
	at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:421)
	at org.elasticsearch.action.search.SearchPhaseController.reducedScrollQueryPhase(SearchPhaseController.java:412)

[droberts195]

@jimczi please could you check if the problem in
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.6+matrix-java-periodic/ES_BUILD_JAVA=java11,ES_RUNTIME_JAVA=zulu11,nodes=virtual&&linux/38/consoleText is the same?

This failure happened in an ML test which timed out because it took more than 20 minutes. However, the root cause is that an assertion tripped while initializing a scroll:

ERROR   0.00s J3 | TooManyJobsIT (suite) <<< FAILURES!
   > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 1200000 msec).
   >    at __randomizedtesting.SeedInfo.seed([ACEE7CCD596311C5]:0)
   > Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1882, name=elasticsearch[node_t1][search][T#3], state=RUNNABLE, group=TGRP-TooManyJobsIT]
   > Caused by: java.lang.AssertionError
   >    at __randomizedtesting.SeedInfo.seed([ACEE7CCD596311C5]:0)
   >    at org.elasticsearch.index.search.stats.ShardSearchStats.lambda$onQueryPhase$2(ShardSearchStats.java:101)
   >    at org.elasticsearch.index.search.stats.ShardSearchStats.computeStats(ShardSearchStats.java:142)
   >    at org.elasticsearch.index.search.stats.ShardSearchStats.onQueryPhase(ShardSearchStats.java:93)
   >    at org.elasticsearch.index.shard.SearchOperationListener$CompositeListener.onQueryPhase(SearchOperationListener.java:155)
   >    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:407)
   >    at org.elasticsearch.search.SearchService.access$100(SearchService.java:126)
   >    at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:360)
   >    at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:356)
   >    at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1117)
   >    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759)
   >    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
   >    at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41)
   >    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
   >    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   >    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   >    at java.base/java.lang.Thread.run(Thread.java:834)

That failure was in 6.6.

If it's a completely different problem and you'd prefer it raised in a new issue let me know.

[jimczi]

If it's a completely different problem and you'd prefer it raised in a new issue let me know.

Yes this is a different issue, the one described here should only happen on master.

[droberts195]

Yes this is a different issue, the one described here should only happen on master

OK cool. I transferred it to #37185.

[davidkyle]

I think this assertion is also causing failures in the mixed cluster test IndexingIT as the error takes down the node.

From the log file:

[2019-01-07T00:42:11,829][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-3] fatal error in thread [elasticsearch[node-3][search][T#24]], exiting
java.lang.AssertionError: null
	at org.elasticsearch.action.search.SearchPhaseController$TopDocsStats.getTotalHits(SearchPhaseController.java:763) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:510) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:421) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.search.SearchPhaseController$1.reduce(SearchPhaseController.java:737) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:101) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.search.FetchSearchPhase.access$000(FetchSearchPhase.java:44) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:86) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=centos/165/console

[jimczi]

IndexingIT is still failing with the same failure:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+bwc-tests/307/console
I am not able to reproduce yet but the previous fix is not enough so reopening.