Add more distance metrics for vector fields

[mayya-sharipova]

• We would like to add more similarity/distance metrics for vectors
• Simplify the API

GET myindex/_vectors/knnsearch
{
  "query": {
  	"match": { "my_text": "text1" }
  },
  "similarity" : "l1norm",
  "field": "my_dense_vector",
  "query_vector": [0.5, 1, -0.5]
}

This is for the simple precise knn search implemented as the sequential scan of documents filtered by the optional query.

Possible similarity metrics: cosine, dotProduct, l1norm, l2norm.
For cosine and dotProduct, a score computed as is. For l1norm and l2norm, the score is computed as 1/l1norm and 1/l2norm.

[elasticmachine]

Pinging @elastic/es-search

[jtibshirani]

I just wanted to leave a couple questions as I’m not able to make the discussion:

• I’m wondering how we’ll support cosine and dotProduct similarities in this API, as they can product negative scores by default.
• Instead of introducing a new _vectors endpoint, would it make sense to add a new query type as we've done in the past? This query type could eventually be extended to allow for approximate NN matching as opposed to operating over all documents.

[mayya-sharipova]

We have met and discussed this issue

I’m wondering how we’ll support cosine and dotProduct similarities in this API, as they can product negative scores by default.

One idea was that that distance metric produce not scores, but sorting fields; and sorting fields can have a negative value. Another idea is to normalize values to make them non-negative (for example, +1 for cosine similarity)

[wmelton]

Is there measurable performance gains by adding this as native functionality vs running the queries as painless routines?

As exciting as it is to see this functionality being added to ES, I'm wondering about extensibility - e.g. what if we want to find the 3 vectors that have the highest relative avg, etc.? Those are obviously custom use cases, but it seems like the true challenge with vector space queries here will be performance.

Using a bitset filter to narrow down the candidate documents first is a good performance hack, but if you're querying against 30 million documents using linear analyzation techniques, will performance be able to remain relatively flat as the corpus grows?

[mayya-sharipova]

@wmelton Thanks for your question.

Currently vectors functions use linear scan over all matched documents. Thus, the overall query time should linearly increase proportionally to the documents' size. We will update the documentation to reflect this.

[mayya-sharipova]

Closing this as l1 and l2 distances have been merged in #44116