Broken highlighting when using char_filter + word_delimiter filter

[lmenezes]

To reproduce the error(using 0.90.0, but was also able to reproduce on 1.0 Beta1):

curl -XPOST http://localhost:9200/foo -d '{ "mappings": { "bar": { "dynamic": "false", "properties": { "id": { "type": "integer" }, "content": { "type": "string", "analyzer": "foobar" } } } }, "settings": { "index": { "analysis": { "char_filter": { "iso_mapping" : { "type" : "mapping", "mappings" : ["ü=>ue"] } }, "filter": { "wordDelimiter": { "type": "word_delimiter", "split_on_numerics": "false", "generate_word_parts": "true", "generate_number_parts": "true", "catenate_words": "true", "catenate_numbers": "true", "catenate_all": "false" } }, "analyzer": { "foobar": { "tokenizer": "whitespace", "filter": [ "lowercase", "wordDelimiter" ], "char_filter": "iso_mapping" } } } } } }'

curl -XPUT http://localhost:9200/foo/bar/1 -d '{ "id": 1, "content": "eins, fünf, sechs" }'
curl -XPUT http://localhost:9200/foo/bar/2 -d '{ "id": 2, "content": "eins, fünf,sechs" }'
curl -XPUT http://localhost:9200/foo/bar/3 -d '{ "id": 3, "content": "eins, vier, sechs" }'
curl -XPUT http://localhost:9200/foo/bar/4 -d '{ "id": 4, "content": "eins, vier,sechs" }'

The, for the broken case(where the char filter is used):

curl -XPOST http://localhost:9200/foo/bar/_search -d' { "from": 0, "size": 300, "query": { "match": { "content": "Fünf" } }, "highlight": { "fields": { "content": { "fragment_size": 50, "number_of_fragments": 5 } } } }'

where we get:

"highlight":{"content":["eins, <em>fünf,</em> sechs"]}}
"highlight":{"content":["eins, <em>fünf,sechs</em>"]}}

And for a working case(no char filter used):

curl -XPOST http://localhost:9200/foo/bar/_search -d' { "from": 0, "size": 300, "query": { "match": { "content": "vier" } }, "highlight": { "fields": { "content": { "fragment_size": 50, "number_of_fragments": 5 } } } }'

where we get:

"highlight":{"content":["eins, vier,sechs"]}}
"highlight":{"content":["eins, vier, sechs"]}}

[lmenezes]

Just noticed that it might no be easy to appreciate the error, but for the broken examples the results for the highlight are
[fünf,] and [fünf,sechs], when it should be only [fünf].
for the working example, only vier is highlighted, as expected.

[s1monw]

this is puzzeling... I am looking into it.

[s1monw]

ok I think I understand what is going on here. The problem is that word_delimiter_filter (btw. the last broken filter in Lucene) is messing up the offsets that are corrected by the char filter ie. whey you pass eins, fünf,sechs through the char_filter and whitespace tokenizer you end up with 2 tokens eins, and fuenf,sechs. The tokenizer already corrected the offset to end at 16 (actually 17 since we did ü -> ue but we correct that since the original value that might be used for highlighting in a stored field / source has length 16) but 'word_delimiter' tries to be smart and checks the chars making it span the entire original token.

The bottom line is 1. don't use word_delimiter if you want reliable highlighting. 2. if you still use it don't use char_filter, you might be able to do this with yet another filter but from the top of my head this seems not straight forward. I can totally see how a TokenFilter that does the same as the char_filter here could be useful.

[lmenezes]

@s1monw guess We should not expect a fix on lucene level for the word_delimiter, right?
also, just so that we don't end up building a tokenfilter as a plugin that does the same as the char_filter, could something like that be added in ES in the near future? I guess it could be pretty useful for cases like that for a lot of people

[s1monw]

@lmenezes I think we have a couple of options here.

• add a token filter that does the same thing as the mapping_char_filter - this seems like a good one to me since I guess lots of users might have a need for it ie. diacritics removal / mapping or folding filters like you do where some filters don't to the right thing like ascii folding ü -> u
• fix word delimiter filter to not mess with offsets.
• make work delimiter filter a tokenizer and support offsets correctly.

I kind of like all of these options while in the meanwhile I am pretty much done with word_delimiter that I'd wanna deprecate it entirely and make it a tokenizer instead to actually fix the root of the problem. What do you think?

[clintongormley]

I know that @jpountz has it on his list of things to do: https://issues.apache.org/jira/browse/LUCENE-5111

[lmenezes]

@s1monw

everything sounds good, but "make work delimiter filter a tokenizer and support offsets correctly" wouldn't really solve the problem here. you use word_delimiter as a tokenizer and you end up with "one two" as a single token, unless there is something here that i don't see.

[lmenezes]

@clintongormley cool :)
but having a token filter that does the same as the char_filter isn't a bad idea anyway.

[lmenezes]

sorry about using this issue to get attention to another issue, but what about #3505 ?
Even though I can totally live without it, since we are going back to normal highlighter anyway...

[lmenezes]

do we have any update on this?

[s1monw]

I don't think we will get to fix the root of the problem here any time soon. I think the only option is to not use word delimiter together with highlighting until it is fixed.