Upgrade to lucene-9.0.0-snapshot-32a0a16aff0

[romseygeek]

This commit upgrades elasticsearch 8.0 to use a snapshot of
Lucene 9.0.0.

[elasticmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[romseygeek]

There are a few nocommits, but surprisingly few. Everything compiles.

Still to do:

• I have silently remapped the SIMPLEFS store type to just use NIOFS. I'm not sure if this is correct and we will just deprecate the option, or if it should throw an error.
• The Version 7.14 constant has the wrong lucene version on it, because we haven't released 8.9 yet
• There are a bunch of classes around segment memory accounting that I have left in but are now essentially redundant, eg RamAccountingRefreshListener and SegmentStats. Segment itself also now has unused memoryInBytes and ramTree fields, which I am guessing should be removed but will need some careful BWC handling.
• CuckooFilter uses PackedInts.getReader() and save(), both of which are now gone. Again this was being used as part of the Streamable impl and so will need BWC to be addressed when this is reimplemented.

[dnhatn]

I have silently remapped the SIMPLEFS store type to just use NIOFS. I'm not sure if this is correct and we will just deprecate the option, or if it should throw an error.

I think we should deprecate the simplefs store option in 7.x and remove it in 8.x.

[iverase]

Hey @romseygeek,

I have been looking into the CuckooFilter issue. I think we should have our or version of PackedInts.Mutable, but before anything we need to improve our bwc testing for that part of the code. It seems it is never exercise so we can easily introduce a bug: #73585

[dnhatn]

@romseygeek We need to build a new Lucene 9.0 snapshot as the current snapshot doesn't have the version 8.9.0, which is being used in 7.14 and 7.15.

[iverase]

LGTM. I have only reviewed the bits affected for the change on endianness of Lucene directory API. I think we should revisit the strategy on KeyStoreWrapper and MetadataStateFormat but it can be done as a follow up.

[jtibshirani]

I looked through with a focus on the new vector format. I left minor comments and also noticed we need to update LuceneFilesExtensions to contain the vector index file .vex.

It'd also be good to extend the disk usage analyzer to cover vectors, but this seems best as a follow-up.

[jtibshirani]

Seems best to throw UnsupportedOperationException here.

[jtibshirani]

Do we need this new private class when we already have one in org.elasticsearch.search.internal?

[dnhatn]

I've reviewed all files and I have a question, but changes look great.

[dnhatn]

Can we reduce the scope of this permission?

[romseygeek]

Not easily, because it's needed by the PluginsService (which used to farm this requirement out to lucene but can't any longer because lucene has removed the class). @rjernst and I have talked very briefly about it, and I see that you've added it to the follow-up list on #74057

[romseygeek]

I looked through with a focus on the new vector format. I left minor comments and also noticed we need to update LuceneFilesExtensions to contain the vector index file .vex.

It'd also be good to extend the disk usage analyzer to cover vectors, but this seems best as a follow-up.

I think both of these come under the disk usage and field usage actions in #74057