Wrap stacked tokens in `match` query in a BlendedTerms query for better scoring

[clintongormley]

Stacked tokens (tokens in the same position) in a match query usually represent alternatives, eg query-time synonym expansion, fuzzy terms, etc

These queries tend to favour the rarer terms, which (esp with fuzzy queries) is likely to be the wrong choice (see #5883 and #3125).

From #8352 (comment)

The BlendedTermQuery should be used whenever two query terms are synonyms of each other and should be treated as 'one thing'. It tries to adjust statistics independently of the scoring function (which may have no concept of IDF) to deal with the problem.

But I think for it to work, it would need per-term boost support? Then we need a rewrite method that can build this instead of BooleanQuery, it would look a lot like the boolean one: https://github.com/apache/lucene-solr/blob/trunk/lucene/core/src/java/org/apache/lucene/search/MultiTermQuery.java#L140

Per-term boost support is required to be able to take the fuzzy edit distance into account.

[clintongormley]

@markharwood please could you take a look at this one

[markharwood]

Being a fundamental ranking issue, I think this should be addressed upstream in Lucene.
I have a Lucene patch in progress that is adding a new MultiTermQuery rewrite method. It contains the logic to balance factors like IDF when there are many (typically automated) expansions of user criteria. We need to overcome Lucene's natural bias towards ranking the rare variants top in these cases and go for a more balanced use of IDF.

Once we've redefined what core features Lucene offers we can work through the implications for the elasticsearch functionality that wraps it.

[clintongormley]

@markharwood +1

[whythecode]

+1

[jeantil]

@clintongormley @markharwood while checking up on the status for #10391 I noticed it was merged and thus FLT is not available anymore in 2.0.0.
The lucene patch seems to have been accepted does it mean this issue is resolved and we can hope to have a ranking on the normal fuzzy query which doesn't rank typos over exact matches ?

Once we've redefined what core features Lucene offers we can work through the implications for the elasticsearch functionality that wraps it.

This sentence in particular suggests there may be more work to be done on the ES side before the ranking is actually "fixed"

[markharwood]

(Lucene) fuzzy queries should rank sensibly now in 2.0. However I'm unclear on the behaviour of match query and synonyms so I would not suggest closing this issue without further investigation.

[ptgamr]

+1

Any updates for this issue?