No highlights returned using dis_max with multi_match queries

[v1r]

Elasticsearch version: 5.4.0 and 5.4.1
Regression: Yes, works fine in ES 2.x
Plugins installed: []

JVM version: 1.8

OS version : Windows 7

Description of the problem:

No highlights returned when searching using dis_max with a multi_match queries.
Elasticsearch fails with 'null_pointer_exception'

Steps to reproduce:

Create the following index

PUT blog 
{
  "mappings": {
    "blogpost": { 
      "properties": { 
        "title":    { "type": "text", "term_vector": "with_positions_offsets"}, 
        "body":     { "type": "text", "term_vector": "with_positions_offsets"  }
      }
    }
  }
}

Add a new document

PUT blog/blogpost/1
{
  "title": "welcome test",
  "body": "foo"
}

Run the following query to search 'test' string.

GET blog/_search
{
  "query": {
    "bool": {
      "must": {
        "dis_max": {
          "queries": [
            {
              "multi_match": {
                "fields": [
                  "title"
                ], 
                "slop": 0, 
                "type": "phrase_prefix", 
                "max_expansions": 10, 
                "query": "test"
              }
            }, 
            {
              "multi_match": {
                "fields": [
                  "body"
                ], 
                "slop": 0, 
                "type": "phrase_prefix", 
                "max_expansions": 10, 
                "query": "test"
              }
            }]
        }
      }
    }
  },
  "highlight": {
    "fields": {
      "title": {
        "number_of_fragments": 0, 
        "matched_fields": [
          "title",
          "body"
        ], 
        "type": "fvh"
      }
    }, 
    "require_field_match": false
  }
}

Result:

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 4,
    "failed": 1,
    "failures": [
      {
        "shard": 3,
        "index": "blog",
        "node": "eMbzhELBQSO9kiRwVss65A",
        "reason": {
          "type": "null_pointer_exception",
          "reason": null
        }
      }
    ]
  },
  "hits": {
    "total": 1,
    "max_score": 0.25811607,
    "hits": []
  }
}

Now, let's update the same document and replace 'foo' with 'test' in the body attribute:

PUT blog/blogpost/1
{
  "title": "welcome test",
  "body": "test"
}

Re-execute the search query above, it will successfully return the result

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 0.2876821,
    "hits": [
      {
        "_index": "blog",
        "_type": "blogpost",
        "_id": "1",
        "_score": 0.2876821,
        "_source": {
          "title": "welcome test",
          "body": " test"
        },
        "highlight": {
          "title": [
            "w<em>elco</em>me <em>test</em>"
          ]
        }
      }
    ]
  }
}

As you can see, the query will always fail if the document does not find a match in both attributes.

--Cheers!

[markharwood]

Highlighting is an inexact science and a highlighter has to balance the competing demands of speed, accuracy and ability to summarize lengthy docs. The reason we have so many highlighter implementations reflects the legacy of various people taking up the challenge of creating the definitive Lucene highlighter.
Each implementation has its pros and cons (preserves phrases, works with n-grams, splits on sentences, is fast etc). A failure to highlight is not necessarily a "bug" but sometimes a deliberate design trade-off of an implementation (speed over accuracy).

What you are seeing is a deficiency with the fvh highlighter and I note that if you set the type to plain your example works OK.

The latest hot contender for one-highlighter-to-rule-them-all is the unified highlighter but I see that this also fails on this example.

I don't know for sure that the failure in this case is an oversight or a deliberate limitation of fvh in trying to be a Fast Vector Highlighter. The plain highlighter offers a workaround in this case and so this issue may ultimately be closed with "wontfix".

[clintongormley]

Given that we're wanting to replace the plain highlighter with the unified highlighter, would be good to fix this.

The unified highlighter dies with

      "type": "illegal_argument_exception",
      "reason": "Less than 2 subSpans.size():1"

Changing the phrase_prefix query to a phrase query works. Makes me wonder if there is something wrong with that query in particular.

@jimczi any thoughts?

[jimczi]

The unified highlighter has a bug when the phrase_prefix query contain a single term. I'll work on a fix but I don't think this is related with the bug in the fvh.

[v1r]

Thank you guys for your input. Please also note that the following query fails too while both queries works in ES 2.x.

GET blog/_search
{
  "query": {
    "bool": {
      "should": [
        {        "match_phrase_prefix" : {
            "title" : "test"
        }
          
      }, {        "match_phrase_prefix" : {
            "body" : "test"
        }
      }
      ]
    }
  },
  "highlight": {
    "fields": {
      "title": {
        "number_of_fragments": 0, 
        "matched_fields": [
          "title",
          "body"
        ], 
        "type": "fvh"
      }
    }, 
    "require_field_match": false
  }
}

@markharwood Thanks for your comment, this is a regression compared to ES 2.x.

Update
Another attempt with the following:

GET blog/_search
{
  "query": {
    "bool": {
      "should": [
        {        
            "match_phrase_prefix" : {
            "title" : "test"
        }},   {        
            "match_phrase_prefix" : {
            "body" : "test"
        }
      }
      ]
    }
  },
  "highlight": {
    "fields": {
      "title": {
        "number_of_fragments": 0, 
        "type": "fvh"
      },
      "body": {
        "number_of_fragments": 0, 
        "type": "fvh"
      }
    }, 
    "require_field_match": false
  }
}

[v1r]

Another simpler query

GET blog/_search
{
  "query": {
    "match_all": {}
  },
  "highlight": {
    "fields": {
      "title": {
        "number_of_fragments": 0, 
        "highlight_query": {
              "match_phrase_prefix" : {
                  "title" : "tests"
              }
        },
        "type": "fvh"
      }
    }, 
    "require_field_match": false
  }
}