Skip to content

Add l1norm and l2norm distances for vectors #44116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jul 11, 2019
Merged

Add l1norm and l2norm distances for vectors #44116

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
eb48e80
Add l1norm and l2norm distances for vectors
mayya-sharipova Jul 8, 2019
670159f
Address Christoph's feedback
mayya-sharipova Jul 10, 2019
13b6e77
Made examples more realistic
mayya-sharipova Jul 11, 2019
2874dbf
Merge remote-tracking branch 'upstream/master' into l1l2-vector-dista…
mayya-sharipova Jul 11, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
142 changes: 4 additions & 138 deletions docs/reference/query-dsl/script-score-query.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ a function to be used to compute a new score for each document returned
by the query. For more information on scripting see
<<modules-scripting, scripting documentation>>.


Here is an example of using `script_score` to assign each matched document
a score equal to the number of likes divided by 10:

Expand All @@ -32,7 +31,6 @@ GET /_search
}
--------------------------------------------------
// CONSOLE
// TEST[setup:twitter]

NOTE: The values returned from `script_score` cannot be negative. In general,
Lucene requires the scores produced by queries to be non-negative in order to
Expand Down Expand Up @@ -76,140 +74,6 @@ to be the most efficient by using the internal mechanisms.
--------------------------------------------------
// NOTCONSOLE

[role="xpack"]
[testenv="basic"]
[[vector-functions]]
===== Functions for vector fields

experimental[]

These functions are used for
for <<dense-vector,`dense_vector`>> and
<<sparse-vector,`sparse_vector`>> fields.

NOTE: During vector functions' calculation, all matched documents are
linearly scanned. Thus, expect the query time grow linearly
with the number of matched documents. For this reason, we recommend
to limit the number of matched documents with a `query` parameter.

For dense_vector fields, `cosineSimilarity` calculates the measure of
cosine similarity between a given query vector and document vectors.

[source,js]
--------------------------------------------------
{
"query": {
"script_score": {
"query": {
"match_all": {}
},
"script": {
"source": "cosineSimilarity(params.query_vector, doc['my_dense_vector']) + 1.0", <1>
"params": {
"query_vector": [4, 3.4, -0.2] <2>
}
}
}
}
}
--------------------------------------------------
// NOTCONSOLE
<1> The script adds 1.0 to the cosine similarity to prevent the score from being negative.
<2> To take advantage of the script optimizations, provide a query vector as a script parameter.

Similarly, for sparse_vector fields, `cosineSimilaritySparse` calculates cosine similarity
between a given query vector and document vectors.

[source,js]
--------------------------------------------------
{
"query": {
"script_score": {
"query": {
"match_all": {}
},
"script": {
"source": "cosineSimilaritySparse(params.query_vector, doc['my_sparse_vector']) + 1.0",
"params": {
"query_vector": {"2": 0.5, "10" : 111.3, "50": -1.3, "113": 14.8, "4545": 156.0}
}
}
}
}
}
--------------------------------------------------
// NOTCONSOLE

For dense_vector fields, `dotProduct` calculates the measure of
dot product between a given query vector and document vectors.

[source,js]
--------------------------------------------------
{
"query": {
"script_score": {
"query": {
"match_all": {}
},
"script": {
"source": """
double value = dotProduct(params.query_vector, doc['my_vector']);
return sigmoid(1, Math.E, -value); <1>
""",
"params": {
"query_vector": [4, 3.4, -0.2]
}
}
}
}
}
--------------------------------------------------
// NOTCONSOLE

<1> Using the standard sigmoid function prevents scores from being negative.

Similarly, for sparse_vector fields, `dotProductSparse` calculates dot product
between a given query vector and document vectors.

[source,js]
--------------------------------------------------
{
"query": {
"script_score": {
"query": {
"match_all": {}
},
"script": {
"source": """
double value = dotProductSparse(params.query_vector, doc['my_sparse_vector']);
return sigmoid(1, Math.E, -value);
""",
"params": {
"query_vector": {"2": 0.5, "10" : 111.3, "50": -1.3, "113": 14.8, "4545": 156.0}
}
}
}
}
}
--------------------------------------------------
// NOTCONSOLE

NOTE: If a document doesn't have a value for a vector field on which
a vector function is executed, an error will be thrown.

You can check if a document has a value for the field `my_vector` by
`doc['my_vector'].size() == 0`. Your overall script can look like this:

[source,js]
--------------------------------------------------
"source": "doc['my_vector'].size() == 0 ? 0 : cosineSimilarity(params.queryVector, doc['my_vector'])"
--------------------------------------------------
// NOTCONSOLE

NOTE: If a document's dense vector field has a number of dimensions
different from the query's vector, an error will be thrown.


[[random-score-function]]
===== Random score function
`random_score` function generates scores that are uniformly distributed
Expand Down Expand Up @@ -323,6 +187,9 @@ You can read more about decay functions
NOTE: Decay functions on dates are limited to dates in the default format
and default time zone. Also calculations with `now` are not supported.

===== Functions for vector fields
<<vector-functions, Functions for vector fields>> are accessible through
`script_score` query.

==== Faster alternatives
Script Score Query calculates the score for every hit (matching document).
Expand Down Expand Up @@ -422,5 +289,4 @@ through a script:
Script Score query has equivalent <<decay-functions, decay functions>>
that can be used in script.



include::{es-repo-dir}/vectors/vector-functions.asciidoc[]
Loading