ES 5.2.2 Heap spikes on aggregations, causing OOM

[vineet01]

The issue is similar to the one reported in #23868.

Tested on oracle JDK 1.8.0_112, 1.8.0_91 with mock data loaded in both 2.3.3 and 5.2.2 clusters running on the same system.
Mappings and doc counts are exactly same for both instances. (62K docs).

Aggregations are being performed on raw fields which are not_analyzed.

Configuration:

• 4GB Max heap size.
• indices.breaker.request.limit: 40%
• Default CMS settings.
• No extra plugins.

Screenshot of Kibana with a visualization which I tested with. (sole purpose of this visualization is to compare performance b/w 2.3.3 and 5.2.2):

https://www.dropbox.com/s/rl5sfucdmo006pe/kibana_es2.3.3.png?dl=0
https://www.dropbox.com/s/r5cqwe19abz3g92/kibana_es5.2.2.png?dl=0

53 MB bzipped heap dump can be downloaded @ https://www.dropbox.com/s/os0bgn4nuq7ra7z/java_pid18280.hprof.bz2?dl=0.

Exception:

Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [request] Data too large, data for [<reused_arrays>] would be larger than limit of [1704040857/1.5gb]

Also on the same note, I will like to share that reducing precision_threshold improves this behaviour, I was able to successfully get results on the same with -> {"precision_threshold":1}.

Please review.

[vineet01]

Another heap dump with a 1 day query in 5.2.2 on a time-series index with 6771187 docs.
No OOM/GC on the same in 2.3.3.

https://www.dropbox.com/s/xxx5qopk51vhuly/java_pid8504.hprof.bz2?dl=0

PS: Same worked as expected in 5.2.2 with {"precision_threshold":1} without any issues.

[jpountz]

This looks like a side-effect of #19215. I don't think there is anything to be fixed here, you are just running a very costly aggregation: the fact that you sort on the value of the cardinality aggregation forces Elasticsearch to compute the cardinality for every bucket. And since we changed the default to be more accurate, this uses more memory than before. I would just recommend that you set the precision_threshold to a low value (eg. 100) explicitly.

[vineet01]

As mentioned above, would that also apply to short range queries (1 day) ?
I get same result on a 1 day query performed on a time-series index (on basis of @timestamp) field.

Even with {"precision_threshold":1}, it only works on small ranges, with multiple visualizations on a dashboard it crashes again.

[almico]

I think the real issue here is the circuit breaker.
On a use case I have, Elasticsearch dies after a single cardinality aggregation request.