Add minhash token filter #20345
Description
It would be nice to be able to use the minhash token filter, using minHash token filter added to Lucene.
The minhash algorithm implements locality-sensitive hashing. Basically, every token is hashed into a hash table. The hash table is then sampled to generate a smaller set of representative tokens.
This has several applications
- Dimensionality reduction that preserves information that can be used in similarity calculations. For example, efficient deduplication over large sets of text, more like this, etc.