[DOCS] Reformat word_delimiter_graph token filter

[jrodewig]

Makes the following changes to the word_delimiter_graph token filter docs:

• Updates the Lucene experimental admonition.
• Updates description
• Adds detailed analyze snippet
• Adds custom analyzer and custom filter snippets
• Reorganizes and updates parameter documentation
• Expands and updates section re: differences between word_delimiter and word_delimiter_graph

Also updates the trim filter docs to note that the trim filter does not change token offsets. Moved to #53220

Preview

http://elasticsearch_53170.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/analysis-word-delimiter-graph-tokenfilter.html

[elasticmachine]

Pinging @elastic/es-search (:Search/Analysis)

[elasticmachine]

Pinging @elastic/es-docs (>docs)

[romseygeek]

I left a bit of a rambly comment on when to use this filter. It's very easy to misuse it - @jimczi may have an opinion here as well.

[romseygeek]

I don't think this is true? catenate_X parameters break phrase searching in general - for example, searching for the exact phrase the wifi is enabled won't match against the token stream above because fi introduces an extra position, so is is indexes as if it were two positions away from wifi. This is a hard problem in lucene - we don't want to start indexing position lengths because that will make phrase queries much slower.

The advantage of the _graph variant is that it produces graphs which can be used at query time to generate several queries, so a query for the wi-fi is enabled will produce two phrase queries, the wi fi is enabled and the wifi is enabled. All good if you've indexed the phrase the wi-fi is enabled, as the first query will match. However, searching for the wifi is enabled won't match - it's all lowercase in the query, so the filter doesn't recognise the need to break it up, and in the index wifi is two positions away from is.

Breaking up words with hyphens in is tricky because of the possibility that people will try and search for the word without the hyphen; I think these are probably better dealt with via synonyms. A better usecase for removing punctuation is for things like part numbers, where you only really want phrase searching within a multi-part token, so you use WDGF with a keyword tokenizer.

[jrodewig]

Aha! The query time bit makes sense to me. I can also add some warnings to the catenate parameters so users know they'll break phrase searches. I'll also amend the intro a bit to cover the product/part number use case.

[jrodewig]

Thanks again for your feedback @romseygeek.

I've made some adjustments throughout the page. Here are the changes:

• Updated the examples in the description to better fit identifiers, such as part numbers
• Added a tip to the description re: identifiers
• Updated the analyze example to fit identifiers
• Added a warning re: indexing and multi-position tokens to the catenate_* and preserve_original parms
• Added a warning re: match_phrase to the catenate_* parms
• Rewrote the differences section to focus on token graph output, including some token graph diagrams.

To better check out the diagrams, you can use this preview link:
http://elasticsearch_53170.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/analysis-word-delimiter-graph-tokenfilter.html

[romseygeek]

Thanks @jrodewig , this looks much better - the token graph diagrams especially are very helpful.

[jrodewig]

Thanks @romseygeek!

[jrodewig]

Backport commits

master 1c8ab01
7.x 28cb4a1
7.6 8e3c1cb