Expose the new SimplePatternTokenizer and SimplePatternSplitTokenizer

[mikemccand]

These tokenizers use Lucene's fast DFA implementation for creating tokens, and are new in 6.5.0 (and specifically in the last upgrade: #23362)

I won't have time to work on this but wanted to get the issue up so we don't forget.

[firstprayer]

What does "exposing" mean? My understanding is exposing them as those in https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-tokenizers.html, basically allowing users to specify these 2 as the tokenize for their documents, but I'm not sure - can you clarify?

Also - I'm very interested in taking up this one but I'm still new to ES. Any code pointer as starting point would be nice :)

[jpountz]

@firstprayer yes, this is what it means. Maybe you can look at how other analysis components are exposed (grepping for their name through the code base should help)?

[glefloch]

@mikemccand can you explain me the difference between SimplePatternTokenizer and SimplePatternSplitTokenizer. The lucene javadoc of SimplePatternSplitTokenizer says:

// TODO: this is copied from SimplePatternTokenizer, but there are subtle differences e.g. we track sepUpto an tokenUpto;
// find a clean way to share it:

After having a look at the code, there already is a PatternTokenizer which run a regexp. In the documentation it says that the new implementation a faster but a bit more limited.

Should we replace PatternTokenizer with SimplePatternTokenizer or SimplePatternSplitTokenizer ?
or should we expose all of them despite they almost do the same things.

[andyb-elastic]

@nik9000 recommended I take this one

@glefloch I think the main functional difference here is that SimplePatternTokenizer returns tokens that match the pattern, where SimplePatternSplitTokenizer splits the input at the places where the pattern matches, but doesn't return the matches themselves. PatternTokenizer can do either depending on how you configure it, there are some examples in its page in the docs

I think we should expose all three, because deprecating PatternTokenizer seems like it would cause major frustration to users who use regex features that aren't supported by the Simple tokenizers. If using PatternTokenizer with runaway regexes is a common pitfall for users then maybe it's worth deprecating.

This also possibly revealed another issue: it looks like the common tokenizers' factories are all being instantiated when an index is created, regardless of if you configure the index to use them. If I register a configurable tokenizer factory that doesn't set default parameters (for example, imagine PatternTokenizerFactory didn't set a default for its pattern param) then seemingly any operation that creates an index will fail when it tries to construct that factory without a default. This doesn't seem like the intended behavior; if this tokenizer isn't used, it seems like its factory shouldn't be constructed.

We're not sure if that's a real problem, I'll dig a little more into it. Assuming it is, I think the best thing to do until that's fixed is set the empty string as the default pattern for both Simple tokenizers. This causes SimplePatternTokenizer to return no tokens, and SimplePatternSplitTokenizer to return the whole input as one token.

[glefloch]

@andy-elastic tanks for the explanation!

[andyb-elastic]

This is merged in master