CI: SmokeTestReindexWithPainlessClientYamlTestSuiteIT reindex/10_script/Modify a document based on id failure

[cbuescher]

This failed two times today so far:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=fedora/738
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.2+multijob-unix-compatibility/os=fedora/125

Error looks like this in the logs:

1> [2017-03-22T08:11:56,595][INFO ][o.e.s.SmokeTestReindexWithPainlessClientYamlTestSuiteIT] [test {yaml=reindex/10_script/Modify a document based on id}]: after test
  1> [2017-03-22T08:11:56,613][INFO ][o.e.s.SmokeTestReindexWithPainlessClientYamlTestSuiteIT] Stash dump on failure [{
  1>   "stash" : {
  1>     "body" : {
  1>       "took" : 397,
  1>       "timed_out" : false,
  1>       "total" : 1,
  1>       "updated" : 0,
  1>       "created" : 1,
  1>       "deleted" : 0,
  1>       "batches" : 1,
  1>       "version_conflicts" : 0,
  1>       "noops" : 0,
  1>       "retries" : {
  1>         "bulk" : 0,
  1>         "search" : 0
  1>       },
  1>       "throttled_millis" : 0,
  1>       "requests_per_second" : -1.0,
  1>       "throttled_until_millis" : 0,
  1>       "failures" : [ ]
  1>     }
  1>   }
  1> }]
FAILURE 1.89s | SmokeTestReindexWithPainlessClientYamlTestSuiteIT.test {yaml=reindex/10_script/Modify a document based on id} <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: Failure at [reindex/10_script:63]: created didn't match the expected value:
   >                        created: expected [2] but was [1]
   > 	at __randomizedtesting.SeedInfo.seed([B20684A64E72811C:3A52BB7CE08EECE4]:0)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:374)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:354)
   > 	at java.lang.Thread.run(Thread.java:745)
   > Caused by: java.lang.AssertionError: created didn't match the expected value:
   >                        created: expected [2] but was [1]
   > 	at org.elasticsearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:92)
   > 	at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:76)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:370)
   > 	... 38 more

Unfortunately the reproduction doesn't work so far locally, here it is anyway:

gradle :qa:smoke-test-reindex-with-painless:integTestRunner -Dtests.seed=7230AEED34CAE77D -Dtests.class=org.elasticsearch.smoketest.SmokeTestReindexWithPainlessClientYamlTestSuiteIT -Dtests.method="test {yaml=reindex/10_script/Modify a document based on id}" -Dtests.security.manager=true -Dtests.locale=sv-SE -Dtests.timezone=Asia/Singapore

[cbuescher]

@nik9000 maybe you can see what's going on here

[nik9000]

Sure!

[nik9000]

Just getting to this - it doesn't reproduce and doesn't make a whole lot of sense. It looks like the reindex only saw 1 document when it meant to see 2. I'd love to see it fail after #23688.

[nik9000]

And here one is.

[nik9000]

From the CI log:

[2017-03-22T11:32:22,828][DEBUG][o.e.a.s.TransportSearchAction] [node-0] [twitter][4], node[Zpj9dh0mT7Oa70Vkd1i-kw], [P], s[STARTED], a[id=nW0VSi1QS-Widm4CmKJFbw]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[twitter], indicesOptions=IndicesOptions[id=38, ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true], types=[], routing='null', preference='null', requestCache=null, scroll=Scroll{keepAlive=5m}, source={
  "size" : 1000,
  "version" : false,
  "sort" : [
    {
      "_doc" : {
        "order" : "asc"
      }
    }
  ]
}}] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [node-0][127.0.0.1:34305][indices:data/read/search[phase/query]]
Caused by: java.lang.NullPointerException
        at org.elasticsearch.search.DefaultSearchContext.buildFilteredQuery(DefaultSearchContext.java:289) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.search.DefaultSearchContext.preProcess(DefaultSearchContext.java:247) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.search.query.QueryPhase.preProcess(QueryPhase.java:90) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:480) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:444) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:252) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:330) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:327) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:623) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:613) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

[nik9000]

That code looks like:

            BooleanQuery.Builder builder = new BooleanQuery.Builder();
            builder.add(query, Occur.MUST);
            for (Query filter : filters) {
                builder.add(filter, Occur.FILTER);
            }
            return builder.build();   <---- NPE right here

Which doesn't make a whole ton of sense.

[nik9000]

These seem to have gone away?