Add support for selecting percolator query candidate matches containing wildcard / prefix queries #25351
Conversation
martijnvg
added
:Search Relevance/Percolator
martijnvg
force-pushed
the
percolator_wildcard_query_support
branch
from
8c32178 to
e48de2c
Compare
…hes containing wildcard and prefix queries. At index time the percolator tries to extract the longest string that doesn't contain a `?` or `*` from the wildcard expression. At search time each query term is expanded into all possible suffixes and then each suffix is turned in all possible prefixes, this to match with any possible extracted wildcard expression. This can speed evaluating percolator queries containing wildcard queries as without this change a lot of times all these percolator queries need to be evaluated all time irregardless if they have no chance of ever matching.
martijnvg
force-pushed
the
percolator_wildcard_query_support
branch
from
e48de2c to
252fdc2
Compare
|
I'm worried this could lead to very large candidate queries if the input document is not tiny? |
|
@jpountz Good point. Perhaps we can build in a limitation? If we insert a special token in the wildcard query terms field to identify all percolator queries with prefix/wildcard queries. At query time if we detect that we create to many suffix terms (50?) or suffix terms or too long (25?) then we just use the special token instead. |
|
I don't know. My gut feeling is that things can degrade pretty quicly. Even if we only extract substrings of length eg. 4, a token of length 20 in a document would generate 20-4 = 16 underlying terms for the candidate query. My gut feeling is that even simple documents already trigger the creation of non trivial candidate queries. I'm a bit worried of making them even more complex. Maybe the right thing to do is to leave it up to the users? They would just have to use (edge) ngrams in their index analyzers? |
Right, maybe that is better. Also it would be clearer why percolation is slower instead of when the percolator is doing what this PR is doing. I'll add some documentation around this. It does mean that wildcard and prefix queries would need to be substituted with |
|
Closing this PR as it can have negative performance impact. |
3 participants
At index time the percolator tries to extract the longest string that doesn't contain a
?or*from the wildcard expression. At search time each query term is expanded into all possible suffixes and then each suffix is turned in all possible prefixes, this to match with any possible extracted wildcard expression.This can speed evaluating percolator queries containing wildcard queries as without this change a lot of times all these percolator queries need to be evaluated all time irregardless if they have no chance of ever matching.