Should we automatically use the FrozenEngine on cold and frozen tiers?

[jpountz]

FrozenEngine is a variant of InternalEngine that requires data to be read-only and lazily opens an IndexReader when needed. Should we consider automatically using this feature in the cold and frozen tiers? It turns out that the changes we did in order to reduce memory usage of shards had the side-effect of making shards faster to open: opening a force-merged shard containing ~100M samples of metric data takes ~1.5 msec on my machine and allocates ~800kB of memory.

Pros:

• More memory for other operations such as aggregations.
• Lucene segment memory usage can no longer prevent Elasticsearch from using very dense nodes.

Cons:

• Slightly slower searches.
• Slightly more garbage for every search.

We could reduce the impact on search by keeping a cache of the most frequently used shards.

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[ywelsch]

We're already automatically using FrozenEngine on the frozen tier. This is motivated by the fact that we want denser configurations on that tier. The cold tier is just as dense as the warm tier, and the benefit of using FrozenEngine is less clear to me, as the memory savings might not matter that much in the grand scheme of things, while the extra IO might not be negligible (you probably have a fast SSD on your machine, and the data set was nicely force-merged, what about slower spinning disks, and non-force-merged data sets?)

[jpountz]

@ywelsch Data was in my FS cache locally indeed, so the 1ms that I mentioned only accounts for CPU usage, it could become much slower on a cold FS cache.

The scenario I have in mind is someone deploying Elasticsearch on d3en.4xlarge instances (64GB RAM, 112TB disk). Assuming 10GB forced-merged shards that use 1MB heap on average, it gives 112*1024/10 ~= 11,500 shards that use ~11GB heap, only to keep shards open.

I agree that the extra I/O might not be negligible however this feels like it would account for enough memory that it wouldn't be negligible. You mentioned non forced-merged datasets, which are interesting because they are indeed slower to open, but they also consume more heap.

I wonder if doing something like having a cache of the 1,000 most frequently used shards could bring the best of both worlds. Assuming that some shards are queried much more frequently than others, the cache would be quite effective in avoiding to open new shards too frequently, and only keeping 1,000 shards open would only consume ~1GB of heap if we're still assuming ~1MB of heap per shard.

[original-brownbear]

We discussed this in the distributed sync today:
At this point we believe we brought down the memory usage per shard for a normal index considerably relative to when this issue was opened.
Fixes to the overhead of cache instances in particular saved hundreds of kb per per shard. What's left currently as far as overhead per shard goes is mainly the mappers as far as I can tell. This aspect would not be helped by moving to the frozen engine. The only thing that I can see where the frozen engine would save heap the case of a shard that has a larger number of segments (or many fields in the actual data) where Lucene data structures start showing up.
Neither of these issues is within the scope of the distributed team so we figured it's best if the search team revisits this discussion to see if there's still any interest in this.

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[original-brownbear]

Closing this as per the above reasoning, we don't practically see heap issues on Frozen nowadays and heap usage of idle shards has been massively reduced ever since this issue was created as well as in the ~1.5 years since the above answer.