-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Description
preserve_original setting is currently not supported in the EdgeNGramTokenFilter https://www.elastic.co/guide/en/elasticsearch/reference/master/analysis-edgengram-tokenfilter.html#analysis-edgengram-tokenfilter
and there is even TODO comment in the master code of Elasticsearch(as of 25th Apr 2020) to Expose preserve_original as shown in this GitHub code link
https://github.com/elastic/elasticsearch/blob/master/modules/analysis-common/src/main/java/org/elasticsearch/analysis/common/EdgeNGramTokenFilterFactory.java#L66
Elasticsearch version (bin/elasticsearch --version):
8.0.0-SNAPSHOT
Plugins installed: []
N/A
JVM version (java -version):
openjdk 14.0.1 2020-04-14
OpenJDK Runtime Environment (build 14.0.1+7)
OpenJDK 64-Bit Server VM (build 14.0.1+7, mixed mode, sharing)
OS version (uname -a if on a Unix-like system):
Darwin LT6577 19.3.0 Darwin Kernel Version 19.3.0: Thu Jan 9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64 x86_64
Description of the problem including expected versus actual behavior:
Its a feature request and mentioned in the TODO of Elasticsearch master code, if provided preserve original functionality would work with n-gram token filter.
Steps to reproduce:
Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.
1. Delete the existing index with the name preserveoriginal to test this feature.
curl --user elastic:password -XDELETE localhost:9200/preserveoriginal
2. Create a new index with custom analyzer which uses edge-ngram token filter.
curl --user elastic:123456 -X PUT "localhost:9200/preserveoriginal?pretty" -H 'Content-Type: application/json' -d'
{
"settings": {
"max_ngram_diff": 50,
"analysis": {
"filter": {
"edge_ngram_filter": {
"type": "edge_ngram",
"min_gram": 1,
"max_gram": 2
}
},
"analyzer": {
"edge_ngram_analyzer": {
"type": "custom",
"tokenizer": "whitespace",
"filter": [
"lowercase",
"edge_ngram_filter"
]
}
}
}
}
}
'
- Check the tokens generated by ngram_analyzer created in the above step:
curl --user elastic:123456 -X GET "localhost:9200/preserveoriginal/_analyze?pretty" -H 'Content-Type: application/json' -d'
{
"analyzer" : "edge_ngram_analyzer",
"text" : "foo"
}
'
4.The output of the above analyzer API.
{
"tokens" : [
{
"token" : "f",
"start_offset" : 0,
"end_offset" : 3,
"type" : "word",
"position" : 0
},
{
"token" : "fo",
"start_offset" : 0,
"end_offset" : 3,
"type" : "word",
"position" : 0
}
]
}
Pease note foo original token isn't present in the result.