Simple combined_fields query fails with java.lang.IndexOutOfBoundsException

[nemphys]

Elasticsearch version (bin/elasticsearch --version): 7.13.1

Plugins installed: [analysis-icu]

JVM version (java -version): built-in

OS version (uname -a if on a Unix-like system): macOS Big Sur

Having replaced the use of multi_match cross_fields queries with the new combined_fields query, I am now running into a dead-end.

The query is very simple:

GET /index-name/_search
{
  "query": {
    "combined_fields": {
      "query": "μαύρο",
      "operator": "AND",
      "fields": [
        "title",
        "description"
      ]
    }
  }
}

and fails with the following stacktrace:

org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:661) [elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:384) [elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:694) [elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:467) [elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$000(AbstractSearchAsyncAction.java:62) [elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:316) [elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66) [elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48) [elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:400) [elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.transport.TransportService$5.handleException(TransportService.java:738) [elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1283) [elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1392) [elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1366) [elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:50) [elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:45) [elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:40) [elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:77) [elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) [elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) [elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) [elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.13.1.jar:7.13.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?] at java.lang.Thread.run(Thread.java:831) [?:?] Caused by: org.elasticsearch.ElasticsearchException$1 at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:633) ~[elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:382) [elasticsearch-7.13.1.jar:7.13.1] ... 22 more Caused by: java.lang.IndexOutOfBoundsException at java.nio.Buffer.checkIndex(Buffer.java:742) ~[?:?] at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:297) ~[?:?] at org.apache.lucene.store.ByteBufferGuard.getByte(ByteBufferGuard.java:119) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27] at org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl.readByte(ByteBufferIndexInput.java:434) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27] at org.apache.lucene.codecs.lucene80.Lucene80NormsProducer$8.longValue(Lucene80NormsProducer.java:397) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27] at org.apache.lucene.search.XMultiNormsLeafSimScorer$MultiFieldNormValues.advanceExact(XMultiNormsLeafSimScorer.java:135) ~[elasticsearch-7.13.1.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27] at org.apache.lucene.search.XMultiNormsLeafSimScorer.getNormValue(XMultiNormsLeafSimScorer.java:91) ~[elasticsearch-7.13.1.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27] at org.apache.lucene.search.XMultiNormsLeafSimScorer.score(XMultiNormsLeafSimScorer.java:103) ~[elasticsearch-7.13.1.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27] at org.apache.lucene.search.XCombinedFieldQuery$CombinedFieldScorer.score(XCombinedFieldQuery.java:453) ~[elasticsearch-7.13.1.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27] at org.apache.lucene.search.TopScoreDocCollector$SimpleTopScoreDocCollector$1.collect(TopScoreDocCollector.java:76) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27] at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreRange(Weight.java:252) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27] at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:239) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27] at org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:45) ~[elasticsearch-7.13.1.jar:7.13.1] at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27] at org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:187) ~[elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:160) ~[elasticsearch-7.13.1.jar:7.13.1] at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445) ~[lucene-core-8.8.2.jar:8.8.2 a92a05e195b775b30ca410bc0a26e8e79e7b3bfb - mdrob - 2021-04-06 16:33:27] at org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:332) ~[elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:287) ~[elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:139) ~[elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:368) ~[elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:422) ~[elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:394) ~[elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) ~[elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.13.1.jar:7.13.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.13.1.jar:7.13.1] ... 6 more

It is really weird and I cannot seem to spot the cause, since:

1. If I use other (random) queries, the query either succeeds or fails with the same exception
2. If I omit the "title" field, the query succeeds,
3. If I omit the "description" field, the query succeeds

Please note that this is definitely a combined_fields issue, since changing the query type to multi_match cross_fields makes it work properly.

This seems to be a Lucene issue, but I have no further idea.

@jtibshirani I suppose you might be interested in this one

[nemphys]

Just found issues #27131 and #34555, which seem to be related.

[jtibshirani]

@nemphys sorry you are running into issues with the new query, I hope we can get to the bottom of it quickly. To help debug, would you be able to supply your index mapping, as well as an example document? (I assume you aren't able to provide a full reproduction?)

Also do you spot more information about the exception elsewhere in the logs? Often times with IndexOutOfBoundsException we will see the offending index too, for example Caused by: java.lang.IndexOutOfBoundsException: 2147483647.

[jtibshirani]

I have a theory as to the issue: if some documents are missing values for one of the fields, then the query can fail when trying to compute norms. It's indeed a problem in Lucene, I filed an issue here: https://issues.apache.org/jira/browse/LUCENE-9999. Does this apply to your set-up -- are there documents missing either title or description?

This is a really unfortunate bug, because part of why we encouraged users to try out combined_fields is to avoid a bug in cross_fields around this same set-up (where some documents are missing fields). I'll make sure this is resolved in an upcoming 7.13 patch release.

[nemphys]

I think that missing values are indeed the issue: I checked, and out of ~47000 total documents there are ~19000 documents missing a description. I tried reindexing while forcing the value of the description field of the documents that lack it to an empty string and, behold, the problem went away (and came back when I reverted the change, so we definitely found the culprit).

This is why the presence of the description field in the query fields array causes exceptions (the complete query includes other fields, too, but I omitted them trying to find the cause of the issue; if I run the query with all other fields except for the description field, there are no exceptions). Also, it explains why some query strings fail, while others succeed (I suppose the ones that succeed happen to not match any of the documents missing a description).

Is this something you can deal with on the Elasticsearch side, or does it have to be fixed by Lucene?

[jtibshirani]

Is this something you can deal with on the Elasticsearch side, or does it have to be fixed by Lucene?

Ultimately it should be fixed in Lucene. But we have ways of applying small Lucene fixes in Elasticsearch without waiting for a new Lucene release, so we won't be tied to that schedule.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[nemphys]

Ping

[jimczi]

The issue is fixed by #74678. The fix is not merged in Lucene yet but we've decided to release with an early fix in the next patch version of Elasticsearch. We'll merge the long term fix in Lucene and remove our fork in a later release.
Closing this issue now as the bug is fixed. Thanks for reporting @nemphys !