Better cross-field support

[jimczi]

Lucene has a new query in the sandbox called BM25FQuery. It is similar to the BlendedTermQuery that we use for cross-fields search but the new query also merges the document statistics (freq + norm) in a way that preserve the benefits of using BM25 formula (term frequency saturates quickly).
There are some work to be done on the Lucene side to improve the integration of this query:
https://issues.apache.org/jira/browse/LUCENE-8710
https://issues.apache.org/jira/browse/LUCENE-8711
However we could already replace the BlendedTermQuery with BM25F since the main logic is in place and should already improve the ranking of documents when cross-fields mode is used in a query.

[elasticmachine]

Pinging @elastic/es-search

[markharwood]

Would this preserve the small bias cross-fields gives towards the "right" field for a term?
Rather than a complete flattening of DF across all fields cross-fields gives the most-popular field the blended DF but all other fields blended DF +1 to ensure they rank lower. I think that's a subtle but important feature to maintain.

[jimczi]

Would this preserve the small bias cross-fields gives towards the "right" field for a term?

No, the BM25FQuery can deboost any field but there is no extra bias based on the inner frequency of each field. I am not sure why this would be an important feature to maintain, it seems arbitrary to me to pick the "right" field as the most-popular one. What is the benefit ? I think that picking the boost per field should be done based on prior knowledge (title is more important than body) and/or automatic learning. Wee could also automatically deboost all fields except the most popular one but I failed to see why this would be better.

[markharwood]

it seems arbitrary to me to pick the "right" field as the most-popular one. What is the benefit ?

Searching for Mark Harwood across firstname and lastname fields should certainly favour any firstname:Mark over a lastname:Mark. Cross-fields was originally created because in these sorts of scenarios IDF would (annoyingly) ensure exactly the wrong field for a term was ranked highest.

[jimczi]

I agree that in this case lastname should not be preferred because Mark is rare in this context but I don't think we should automatically boost the firstname. With default BM25F a document that contains the term in the firstname field would have the same score than a document that contains the same term in the lastname field. If we start applying automatic boost on top of the default we'll have issues to mix them with the explicit boost that the user can set on individual fields.

[markharwood]

a document that contains the term in the firstname field would have the same score than a document that contains the same term in the lastname field.

That was the problem I was hoping to avoid.

we'll have issues to mix them with the explicit boost that the user can set on individual fields.

We very nearly didn't provide the facility for user-supplied boosts with cross-fields. My preference was to error if supplied.
We need to think about the reasons for having different fields in the first place :

1. Summary/detail - this is the title vs body scenario. Terms do not change their meaning when moved from one field to the other.
2. Semantic context - this is the firstname vs lastname scenario. Terms do change their meaning when moved from one field to the other.

In scenario 1) it is appropriate for an elastic admin to say "boost field X over Y for all queries".
In scenario 2 it is more appropriate for the terms in each user query to be considered by the cross-fields scoring on a case-by-case basis, determining what is the correct field context and biasing matches towards that.
The only reason we kept support for any admin-supplied field boosts in cross-fields was where the admin was (ab)using cross-fields' flattening of IDF while tackling scenario 1). This muddied the waters in my view. For me cross-fields was always about scenario 2.

[jimczi]

Ok thanks for explaining. IMO BM25F is really about 1) and I wonder if 2) could be tackled differently. For instance if you have 2 fields, street and city, a query for oxford street would boost oxford in the city field because it seems more popular. There are also counter-examples like your example but I think we should decorrelate these use cases. The semantic context could use BM25F to match documents but I feel like it needs to do more things than just analyzing the idf of individual terms while for 1) BM25F can be use directly.

[markharwood]

I think "cross-fields" could perhaps have been better named as "correct field" - meaning it tries to determine which is the appropriate semantic context for each term and boosts accordingly.

Maybe this "correct field" technique should only apply to single-term fields for now. In your street vs city example the term oxford is further qualified by other terms in the field e.g. road/rd/street/st. If the field only has one term there's no additional context to consider and I suspect the existing approach works well. For multi-term fields I guess we could get into more complex attempts at query-understanding using shingles or phrase queries but that's perhaps for another day.