Support for more numerical types in the dense_vector field

[joshdevins]

Currently the dense_vector field supports only 32-bit float values. At a minimum, the field should also support 8-bit integer values.

Background

There is a growing emphasis in the NLP and neural search landscapes to utilize quantization along with other techniques in order to improve efficiency while trying to maintain effectiveness of methods. This is particularly evident in BERT/Transformer models [1][2] and embeddings used for retrieval and ranking [3]. In order to support more efficient storage of and computation on embeddings, the dense_vector field needs to support a wider range of numeric types.

[1] Q8BERT: Quantized 8Bit BERT
[2] Faster and smaller quantized NLP with Hugging Face and ONNX Runtime
[3] Sentence-Transformers; Model Distillation

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[joshdevins]

Another important use-case where efficiency of storage matters is when we extract token IDs at ingest time, but only use them after ranking for tasks like extractive question answering. In this case, we extract token IDs (tokenization) at ingest time to avoid the tokenization process on documents at query time. The token IDs are fed to a model which extracts an "answer span" — a span of text which answers the question being posed as a query. Since token IDs are typically in the range of just ~30k integers, it would be significantly more efficient to store them using lower precision numerical types, e.g. uint16, int32.

[joshdevins]

Another new approach is learning bit vector representations/embeddings. This is becoming popular both at the sentence/paragraph embedding level [1] as well as the term embedding level [2]. Being able to perform fast Hamming distance ANN would be a natural follow on from this.

[1] Efficient Passage Retrieval with Hashing for Open-domain Question Answering
[2] ColBERT binarization

See also: #48322

[jtibshirani]

In Lucene we're currently discussing the idea of storing each element at lower precision: https://issues.apache.org/jira/browse/LUCENE-10577.

This also popped up recently in a Discuss issue, where a user found kNN was using more memory than the available RAM: https://discuss.elastic.co/t/ann-search-performance-setup/307129.

[jtibshirani]

I'm closing this in favor of two issues:

• Add a field type for high-dimensional bit vectors. #48322
• Support 8-bit integer values in dense_vector #89784

Feel free to file other issues about specific types, or comment on those with feedback.