The Whitespace tokenizer should support the `max_token_length` parameter

[cbuescher]

Other tokenizers (like Standard) support overriding the max_token_length parameter, but it seems Whitespace doesn't, while the underlying Lucene WhitespaceTokenizer seems to support this parameter. We should probably enable setting this parameter in Elasticsearch as well.

[jpountz]

+1 but then we'd need a limit so that users don't set it to something that is unreasonable.

[cbuescher]

The existing restriction in Standard tokenizer allows max_token_length to be up to 1024 * 1024 characters (https://lucene.apache.org/core/6_6_0/core/org/apache/lucene/analysis/standard/StandardTokenizer.html#setMaxTokenLength-int-). Maybe this should also be lower? But hard to pick a number, so should we just leave it to the user to pick a good maximum length? The default should work most of the time.

[jpountz]

Lucene does not support terms that are more than 32766 bytes, and Java chars can be at most 3 bytes so maybe a limit around 10000 would be a good start.