Updating ICU version to the latest when releasing a major ES version

[damienalexandre]

I just discovered that analysis ICU is still at version 54.1 in Elasticsearch 5.0 (same as ES 2.4) as you can see here: https://github.com/elastic/elasticsearch/blob/4b5f87cb7dcea80bca99f863a02e585b8c2c1e17/plugins/analysis-icu/build.gradle

ICU 54 is quite old now and the break iterator, collations and emoji support have been greatly improved since then (two years ago). See http://site.icu-project.org/download.

As updating ICU require a re-indexation, I'm wondering what is the path for power user who wants to benefits from the latest CLDR / Unicode features? I understand that updating Elasticsearch should not require a re-index operation, so I guess I should build my own icu-analysis plugin maybe?
The Lucene ICU components are also not up to date with ICU as they requires the version 56.1 instead of 58.1.

As this plugin depend on a library that is very often updated, maybe it's not the best choice to ship it with Elasticsearch directly? As an external plugin, it could follow the ICU versions without blocking cluster updates for users.

Thanks,
Damien

[clintongormley]

@mikemccand are we out of date with Lucene's ICU? I'm guessing that we can't upgrade now until 6.0?

[mikemccand]

@clintongormley @damienalexandre Yeah, ICU moves fast, which is a good thing (tracking Unicode standards, which move fast, as the world's languages, and emojis (!), are better modeled). ICU 54.1 is oldish (~ 2 years). Lucene's ICU analyzers in 6.3 is based on ICU 56.1, but that's also ~1 year oldish, and ICU is now up to 58.1 currently (released last month). Why doesn't ES's ICU plugin track Lucene's ICU versions?

But that said, analyzer backwards compatibility is tricky, because you really need all indexed documents into an index (and any future searches against that index) to use the same analyzer versions, even if you did a major ES/Lucene version upgrade at some point.

And the Lucene project does not require that changes to analyzer behavior can only happen on major releases. The major release backwards compatibility promise is only about the low-level index format for reading and writing, now how the tokens were created from incoming text. I'll open an issue now to upgrade ICU to 58.1 for Lucene 6.4.

Net/net I don't think ES should make back-compat promises around analysis. Lucene tried this for a while, requiring a Version parameter to all analysis components, but that was too much and we removed it. I think, instead, we should document that if you use the ICU plugin, you must fully re-index, even within a minor release, if the ICU version has changed.

[mikemccand]

I opened https://issues.apache.org/jira/browse/LUCENE-7540 for Lucene to upgrade to ICU 58.1.

[dadoonet]

Why doesn't ES's ICU plugin track Lucene's ICU versions?

May be I'm wrong but I believe that's because of #15238.
Which means to me that we need to manually update deps.

[clintongormley]

thanks @mikemccand

Sounds like we should (a) open a blocker for 6.0 to upgrade ICU and (b) document the need to reindex when using ICU.

Do you think there is any mileage in providing versioned ICU plugins?

[mikemccand]

bq. Sounds like we should (a) open a blocker for 6.0 to upgrade ICU and (b) document the need to reindex when using ICU.

++ to both

bq. Do you think there is any mileage in providing versioned ICU plugins?

I think documenting that you may need to reindex is fine.