Frozen Indices

[s1monw]

Frozen indices are intended to allow much higher ratios of disk storage to heap, at the expense of search latency. The idea is to keep frozen indices in an unloaded state (ie not loading lucene data-structures into heap) until they are searched. When a search targets frozen indices, each index would be searched sequentially (instead of in parallel), and each index would be loaded, searched, then unloaded again. Frozen indices would be replicated, unlike closed indices today. In-fact frozen indices are more like open indices with metadata in memory and data fully on disk until it's needed to be loaded

Frozen indices will be available in the default distribution but not in the pure OSS distribution.

In oder to implement this, the feature is broken down to the following steps:

• Add the ability to force searches to go through a threadpool that has a limited amount of threads to ensure shards data is materialized sequentially or with very limited concurrency. This will allow frozen indices to be marked as index.search.throttled that will force searches as well as other data access like _get or _explain through a dedicated search_throttled threadpool which has only 1 thread by default. (Introduce a search_throttled threadpool #33732)
• Prevent throttled indices to be included in _search and _msearch by default. This will prevent frozen indices to be included in wildcard or alias searches by default but allows opt-in to searching these indices. At the same time these indices are treated like ordinary indices in all other APIs (Prevent throttled indices to be searched through wildcards by default #34354)
• Add a frozen engine that maintains static stats and opens an index reader on demand. The searcher / reader returned form this engine will be able to reset itself after it search phase to release resources and open up a new searcher / reader from the same commit point on subsequent search phases like fetch or scan / scroll (Add a frozen engine implementation #34357)
• Add an Java API that allows to open a closed index as frozen. This is the simplest way to introduce frozen indices, further improvements on how we go from Open to Frozen can be done as followups if necessary. As a start an index must got through Closed in order to be frozen or unfrozen. (Add a java level freeze/unfreeze API #35353)
• Add an optimization that returns a special index reader that allows range queries on date fields to rewrite in can_match phase in order to exclude frozen indices in can_match without opening up their index reader. (Allow efficient can_match phases on frozen indices #35431)
• Add a REST API and Documentation for frozen indices and how to freeze and index (Add a _freeze / _unfreeze API #35592)
• Add high-level REST client support for _freeze / _unfreeze(Add high-level REST client API for _freeze and _unfreeze #35723)

[s1monw]

@debadair where do you see the documentation go for this feature?

[elasticmachine]

Pinging @elastic/es-distributed

[s1monw]

I thought about some future work on this and wanted to put it here for awareness and potential discussion:

• The current design would open a reader before we check the cache for hits and even if we cache we would invalidate the cache after each search phase. I wonder if we can trick the lazy reader design into having it's own cache key helper that we can trick into surviving close calls of the actual reader. That way we might be able to allow request cache hits on frozen indices without actually opening the reader

• should we allow forcemerge on a frozen index? This might allow us to freeze first and the select a new codec to get best compression. It’s relatively straight forward to do with our infrastructure and an IW that is not pulling NRT readers. And we have limited threads doing this anyway in the force merge threadpool. That's totally something we can do way later.