Support kuromoji user dictionary set directly in the settings file

[tatsuya]

It would be nice if kuromoji_tokenizer supports loading user dictionary via array of dictionary entries in the settings json directly, not only from the file.

Current settings example looks like the below:

{
  "settings": {
    "index": {
      "analysis": {
        "tokenizer": {
          "kuromoji_user_dict": {
            "type": "kuromoji_tokenizer",
            "mode": "extended",
            "discard_punctuation": "false",
            "user_dictionary": "userdict_ja.txt"
          }
        },
        "analyzer": {
          "my_analyzer": {
            "type": "custom",
            "tokenizer": "kuromoji_user_dict"
          }
        }
      }
    }
  }
}

My suggestion is to have new json property named user_dictionary_entires (or similar) at the same level of current user_dictionary, and it accepts the array of dictionary entries. If both user_dictionary and user_dictionary_entries given, then it has to either merge both inputs or use only one of them though, I think simply prioritize one of those inputs would be simpler. This is actually pretty similar to the way the Synonym Token Filter supports already.

So the new json format would be:

{
  "settings": {
    "index": {
      "analysis": {
        "tokenizer": {
          "kuromoji_user_dict": {
            "type": "kuromoji_tokenizer",
            "mode": "extended",
            "discard_punctuation": "false",
            "user_dictionary_entires": [
              "東京スカイツリー,東京 スカイツリー,トウキョウ スカイツリー,カスタム名詞",
              "..."
            ]
          }
        },
        "analyzer": {
          "my_analyzer": {
            "type": "custom",
            "tokenizer": "kuromoji_user_dict"
          }
        }
      }
    }
  }
}

If this sounds good to you, I can create a pull request anytime. Thank you!

[tlrx]

@tatsuyaoiw Thanks for your suggestion. I marked this issue as "discuss", meaning it get on hold until some team members knowledgeable in analyzers can make a proper answer :)

[tatsuya]

@tlrx Cool! Looking forward to hear back.

[cbuescher]

@tatsuyaoiw thanks for the suggestion, we discussed this about it sounds reasonable as long as the user dictionary remains relatively small. I don't know how practical this is though, maybe @johtani has some thoughts about this? In any case, can you briefly describe how you are planning to use that feature? Is it only for testing or to extend the tokenizer dictionary in production?

[tatsuya]

@cbuescher Thanks. This should primarily be for testing purpose but in case if it just works with small number of additional dictionary entries it can also be used in production. I think this gives a lot more flexibility because we don't need to have any dependency on the filesystem.

[johtani]

I think user dictionary is not small usually.
We can support it but we should have some limitation for user dictionary size.
And we have to change Lucene's source, because JapaneseTokenizerFactory only support the file of dictionary.

[romseygeek]

cc @elastic/es-search-aggs

[cbuescher]

Re-visting this, the Nori tokenizer now supports custom inline dictionaries (#35842), so we should treat Kuromoji similarly and add this feature there as well (keeping in mind @johtani's remark that we should maybe have some limitation on the dictionary size).