Unified highlighter with bool query containing nested query throws illegal argument exception #106693
Description
Elasticsearch Version
8.10+
Installed Plugins
No response
Java Version
bundled
OS Version
bundled
Problem Description
When running a bool query containing nested query with unified highlighting we sometimes get illegal argument exception.
Query:
{
"query": {
"bool": {
"must": {
"match": {
"subject": {
"query": "bike"
}
}
},
"must_not": {
"nested": {
"query": {
"match": {
"attachments.name": {
"query": "invite"
}
}
},
"path": "attachments"
}
}
}
},
"highlight": {
"require_field_match": true,
"fields": {
"subject": {
"type": "unified"
}
}
}
}
Error:
"type": "illegal_argument_exception",
"reason": "Reader FilterLeafReader(FilterLeafReader(FilterLeafReader(FieldUsageTrackingLeafReader(reader=FilterLeafReader(_gp(8.11.1):C3054/2:[diagnostics={java.runtime.version=11.0.10+9-LTS, mergeFactor=2, os.arch=x86_64, source=merge, java.vendor=Amazon.com Inc., os.version=10.16, mergeMaxNumSegments=-1, java.version=11.0.10, java.vm.version=11.0.10+9-LTS, lucene.version=8.11.1, timestamp=1697821346583, os=Mac OS X}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}]:fieldInfosGen=1:dvGen=1 :softDel=2 :id=9mvdr58lxesb0l6jrs63zxpmt))))) does not support caching"
We have done some investigating and testing:
- We think the issue we are experiencing is related to this change Use the Weight#matches mode for highlighting by default #96068 introduced in 8.10. We have tested with 8.9. and indeed there was no error, we also tested with 8.11.3 and 8.12.2 where we got the error.
- When we change to "plain" highlighter or we set "require_field_match" to false (i.e. we turn off Weight#matches mode) there is no error.
- There was similar problem reported here Universal highlighter with weight_matches_mode throws cryptic error #101667 and resolved here Disable weight_matches when kNN query is present #101713 by disabling Weight#matches mode in that specific case. Two more issues related to this are Unified highlighter fails by default in hybrid search with knn #101141 and Fix hybrid search and unified highlighting #101621 .
Steps to Reproduce
We have tried creating minimal set of steps for reproducing this but we failed. Query posted above is indeed minimal one that produced an error. In the process we learned that query itself is not enough to reproduce it, indexed data set is also relevant. Both fields hit by the query "subject" and "attachments.name" are mapped as following:
"type": "text",
"term_vector": "with_positions_offsets"
Also, if "must_not" part of the query contains a query which hits no documents we don't get the error, only if some documents are hit by that query error appears.
Logs (if relevant)
No response