NullPointerException if percolate query is used inside bool query

[Lumaraf]

Elasticsearch version:
6.2.3

JVM version:
1.8.0_161

OS version (uname -a if on a Unix-like system):
Debian 4.9.30-2+deb9u2

Description of the problem including expected versus actual behavior:
A NullPointerException is thrown if a percolate query is used inside a bool query if other should clauses match, but not the percolate query.

Steps to reproduce:

PUT /percolator_bug
{
  "mappings": {
    "docs": {
      "properties": {
        "a": {
          "type": "keyword"
        },
        "b": {
          "type": "percolator"
        }
      }
    }
  }
}

POST /percolator_bug/docs/1?refresh=true
{
  "a": "test"
}

POST /percolator_bug/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "term": {
            "a": "test"
          }
        },
        {
          "percolate": {
            "field": "b",
            "document": {}
          }
        }
      ]
    }
  }
}

Log:

[2018-04-09T12:00:41,319][WARN ][r.suppressed             ] path: /percolator_bug/_search, params: {index=percolator_bug}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:274) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:132) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:243) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:107) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:49) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:217) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:171) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:49) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:217) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:527) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1098) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1191) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1175) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.action.search.SearchTransportService$6$1.onFailure(SearchTransportService.java:385) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:324) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:318) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:312) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1002) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.3.jar:6.2.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: org.elasticsearch.ElasticsearchException$1
	at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:619) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.action.search.SearchPhaseExecutionException.guessRootCauses(SearchPhaseExecutionException.java:170) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.action.search.SearchPhaseExecutionException.getCause(SearchPhaseExecutionException.java:111) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.apache.logging.log4j.core.impl.ThrowableProxy.<init>(ThrowableProxy.java:139) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.impl.ThrowableProxy.<init>(ThrowableProxy.java:122) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.impl.MutableLogEvent.getThrownProxy(MutableLogEvent.java:326) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.pattern.ExtendedThrowablePatternConverter.format(ExtendedThrowablePatternConverter.java:64) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:38) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.layout.PatternLayout$PatternSerializer.toSerializable(PatternLayout.java:333) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.layout.PatternLayout.toText(PatternLayout.java:232) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:217) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:57) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:177) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:170) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:161) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:448) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:403) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146) ~[log4j-core-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.spi.ExtendedLoggerWrapper.logMessage(ExtendedLoggerWrapper.java:217) ~[log4j-api-2.9.1.jar:2.9.1]
	at org.elasticsearch.common.logging.PrefixLogger.logMessage(PrefixLogger.java:102) ~[elasticsearch-core-6.2.3.jar:6.2.3]
	at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2116) ~[log4j-api-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2100) ~[log4j-api-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1989) ~[log4j-api-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1851) ~[log4j-api-2.9.1.jar:2.9.1]
	at org.apache.logging.log4j.spi.AbstractLogger.warn(AbstractLogger.java:2589) ~[log4j-api-2.9.1.jar:2.9.1]
	at org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:133) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:96) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:91) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:58) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:91) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.raisePhaseFailure(AbstractSearchAsyncAction.java:222) ~[elasticsearch-6.2.3.jar:6.2.3]
�
more
Caused by: java.lang.NullPointerException
	at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:675) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
	at org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:725) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:583) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
	at org.apache.lucene.search.IndexSearcher.searchAfter(IndexSearcher.java:568) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:493) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
	at org.elasticsearch.percolator.PercolatorMatchedSlotSubFetchPhase.hitsExecute(PercolatorMatchedSlotSubFetchPhase.java:90) ~[?:?]
	at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:170) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:376) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:351) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:316) ~[elasticsearch-6.2.3.jar:6.2.3]
	... 9 more

[elasticmachine]

Pinging @elastic/es-search-aggs

[martijnvg]

@Lumaraf Thanks for reporting this bug and sharing an easy way to reproduce it.

What I do wonder is why you are trying to use a percolate query inside a percolator query? Although the query dsl will allow this, we haven't really thought about this usage of the percolate query and as you have seen doesn't work as expected.

[Lumaraf]

I don't have percolate queries stored in percolator fields. For the example mapping i only reduced the fields to the absolute minimum necessary to reproduce the bug. Fields used by the percolator and those stored in the documents are kept separate in my actual use case.

[martijnvg]

@Lumaraf Ok, that makes sense.