Heap Explosion on even small cardinality queries in ES 5.3.1 / Kibana 5.3.1

[jay-dihenkar]

Elasticsearch version: 5.3.1

Plugins installed: [only defaults]

JVM version: java version "1.8.0_112

OS version: centos 6.8 ( 2.6.32-642.6.2.el6.x86_64 )

Description of the problem including expected versus actual behavior:

When loading unique counts on a field of values having 417docs with 417 total records, ES is going OOM. On introducing the circuit breaker I could capture this in Kibana :

Visualize: [request] Data too large, data for [<reused_arrays>] would be [119719468000/111.4gb], which is larger than the limit of [8511422464/7.9gb] [request] Data too large, data for [<reused_arrays>] would be [138553307200/129gb], which is larger than the limit of [8511422464/7.9gb] [request] Data too large, data for [<reused_arrays>] would be [111728258400/104gb], which is larger than the limit of [8511422464/7.9gb] [request] Data too large, data for [<reused_arrays>] would be [117386429600/109.3gb], which is larger than the limit of [8511422464/7.9gb] [request] Data too large, data for [<reused_arrays>] would be [120568676800/112.2gb], which is larger than the limit of [8511422464/7.9gb] [request] Data too large, data for [<reused_arrays>] would be [108346902400/100.9gb], which is larger than the limit of [8511422464/7.9gb] [request] Data too large, data for [<reused_arrays>] would be [106682848800/99.3gb], which is larger than the limit of [8511422464/7.9gb] [request] Data too large, data for [<reused_arrays>] would be [136041769600/126.6gb], which is larger than the limit of [8511422464/7.9gb] [request] Data too large, data for [<reused_arrays>] would be [140788947200/131.1gb], which is larger than the limit of [8511422464/7.9gb]

For 417 Docs is this expected????!!!

• Field is not analyzed
• Docs are flat, no nesting

The query being hit for aggregation in debug log is :

[2017-04-27T02:36:36,042][DEBUG][o.e.a.s.TransportSearchAction] [i-vL8bl] [test-index][4], node[i-vL8blaT026-wCIWYo16g], [P], s[STARTED], a[id=bVc3KluLSg2mCrR3FRiENw]: Fail
ed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[test-index], indicesOptions=IndicesOptions[id=39, ignore_unavailable=true, allow_no_indices=true, expand_
wildcards_open=true, expand_wildcards_closed=false, allow_alisases_to_multiple_indices=true, forbid_closed_indices=true], types=[], routing='null', preference='149327808
1183', requestCache=null, scroll=null, source={
  "size" : 0,
  "query" : {
    "bool" : {
      "must" : [
        {
          "query_string" : {
            "query" : "*",
            "fields" : [ ],
            "use_dis_max" : true,
            "tie_breaker" : 0.0,
            "default_operator" : "or",
            "auto_generate_phrase_queries" : false,
            "max_determinized_states" : 10000,
            "enable_position_increments" : true,
            "fuzziness" : "AUTO",
            "fuzzy_prefix_length" : 0,
            "fuzzy_max_expansions" : 50,
            "phrase_slop" : 0,
            "analyze_wildcard" : true,
            "escape" : false,
            "split_on_whitespace" : true,
            "boost" : 1.0
          }
        },
        {
          "range" : {
            "ES_timestamp" : {
              "from" : 1487183400000,
              "to" : 1487269800000,
              "include_lower" : true,
              "include_upper" : true,
              "format" : "epoch_millis",
              "boost" : 1.0
            }
          }
        }
      ],
      "disable_coord" : false,
      "adjust_pure_negative" : true,
      "boost" : 1.0
    }
  },
  "_source" : {
    "includes" : [ ],
    "excludes" : [ ]
  },
  "aggregations" : {
    "2" : {
      "terms" : {
        "field" : "<fieldname>",
        "size" : 5,
        "min_doc_count" : 1,
        "shard_min_doc_count" : 0,
        "show_term_doc_count_error" : false,
        "order" : [
          {
            "1" : "desc"
          },
          {
            "_term" : "asc"
          }
        ]
      },
      "aggregations" : {
        "1" : {
          "cardinality" : {
            "field" : "<fieldname>"
          }
        }
      }
    }
  }
}}]

Loading the same on precision value of {"precision_threshold": 1 } is working as expected but again fails for threshold of 10.

The mappings for the field are :

"<fieldname>": {
"type": "string",
"index": "not_analyzed",
"fielddata": false
},

PS: I have already gone through the following issues before creating this one to ensure it's not a duplicate!

Issues:
#21942
#19215

And on discuss forum:
https://discuss.elastic.co/t/mapping-change-on-upgrade-to-5-2-2/82622

[danielmitterdorfer]

Hi @jay-dihenkar, would you be able to share your document corpus (maybe anonymized if it contains sensitive data)?

[danielmitterdorfer]

FWIW, here is my attempt to reproduce this issue (tested again Elasticsearch 5.3.1 with default settings) but this example works fine (that's why I'm asking for the document corpus).

[jay-dihenkar]

Hi,

Some points I want to highlight:

• Total docs in index is : 139000
• We have upgraded from 2.3.3 to 5.3.1 so the mappings which I have given are not same as you have mentioned.
• We have 180 fields in the doc, the field on which aggregation is done is 'uuid' sample value: 4015759c-9764-4fee-b735-720dc6b3e28f
• We are having a timestamp field "ES_TIMESTAMP" which is of type date so we're quering the number of docs for 1 day which is 417.
• The output for cat indices is :

health status index                              uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   test-index                      J7kU9LXHSc-7NGNKHmHRbA   9   2     139856            0    195.1mb        195.1mb

I'll further see if I can anonymize the data and share a mock index.

[danielmitterdorfer]

Thanks for the update @jay-dihenkar. @colings86 can you please have a look whether you can explain this behavior?

[colings86]

@jay-dihenkar are you able to do the same request directly to ES (not through kibana) and get the same result? Also in the server logs there should be a stack trace for this exception could you paste that here (including all caused by traces)?

[jay-dihenkar]

Hi,

Putting details::
curl query =>

> curl -XPOST http://ip:port/test-index/_search -d @t
{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[request] Data too large, data for [<reused_arrays>] would be [119240039600/111gb], which is larger than the limit of [8511422464/7.9gb]","bytes_wanted":119240039600,"bytes_limit":8511422464},{"type":"circuit_breaking_exception","reason":"[request] Data too large, data for [<reused_arrays>] would be [137971939600/128.4gb], which is larger than the limit of [8511422464/7.9gb]","bytes_wanted":137971939600,"bytes_limit":8511422464},{"type":"circuit_breaking_exception","reason":"[request] Data too large, data for [<reused_arrays>] would be [111410235200/103.7gb], which is larger than the limit of [8511422464/7.9gb]","bytes_wanted":111410235200,"bytes_limit":8511422464},{"type":"circuit_breaking_exception","reason":"[request] Data too large, data for [<reused_arrays>] would be [116703122800/108.6gb], which is larger than the limit of [8511422464/7.9gb]","bytes_wanted":116703122800,"bytes_limit":8511422464},{"type":"circuit_breaking_exception","reason":"[request] Data too large, data for [<reused_arrays>] would be [120376379200/112.1gb], which is larger than the limit of [8511422464/7.9gb]","bytes_wanted":120376379200,"bytes_limit":8511422464},{"type":"circuit_breaking_exception","reason":"[request] Data too large, data for [<reused_arrays>] would be [107969413200/100.5gb], which is larger than the limit of [8511422464/7.9gb]","bytes_wanted":107969413200,"bytes_limit":8511422464},{"type":"circuit_breaking_exception","reason":"[request] Data too large, data for [<reused_arrays>] would be [106957774000/99.6gb], which is larger than the limit of [8511422464/7.9gb]","bytes_wanted":106957774000,"bytes_limit":8511422464},{"type":"circuit_breaking_exception","reason":"[request] Data too large, data for [<reused_arrays>] would be [136670085200/127.2gb], which is larger than the limit of [8511422464/7.9gb]","bytes_wanted":136670085200,"bytes_limit":8511422464},{"type":"circuit_breaking_exception","reason":"[request] Data too large, data for [<reused_arrays>] would be [140459030800/130.8gb], which is larger than the limit of [8511422464/7.9gb]","bytes_wanted":140459030800,"bytes_limit":8511422464}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"test-index","node":"i-vL8blaT026-wCIWYo16g","reason":{"type":"circuit_breaking_exception","reason":"[request] Data too large, data for [<reused_arrays>] would be [119240039600/111gb], which is larger than the limit of [8511422464/7.9gb]","bytes_wanted":119240039600,"bytes_limit":8511422464}},{"shard":1,"index":"test-index","node":"i-vL8blaT026-wCIWYo16g","reason":{"type":"circuit_breaking_exception","reason":"[request] Data too large, data for [<reused_arrays>] would be [137971939600/128.4gb], which is larger than the limit of [8511422464/7.9gb]","bytes_wanted":137971939600,"bytes_limit":8511422464}},{"shard":2,"index":"test-index","node":"i-vL8blaT026-wCIWYo16g","reason":{"type":"circuit_breaking_exception","reason":"[request] Data too large, data for [<reused_arrays>] would be [111410235200/103.7gb], which is larger than the limit of [8511422464/7.9gb]","bytes_wanted":111410235200,"bytes_limit":8511422464}},{"shard":3,"index":"test-index","node":"i-vL8blaT026-wCIWYo16g","reason":{"type":"circuit_breaking_exception","reason":"[request] Data too large, data for [<reused_arrays>] would be [116703122800/108.6gb], which is larger than the limit of [8511422464/7.9gb]","bytes_wanted":116703122800,"bytes_limit":8511422464}},{"shard":4,"index":"test-index","node":"i-vL8blaT026-wCIWYo16g","reason":{"type":"circuit_breaking_exception","reason":"[request] Data too large, data for [<reused_arrays>] would be [120376379200/112.1gb], which is larger than the limit of [8511422464/7.9gb]","bytes_wanted":120376379200,"bytes_limit":8511422464}},{"shard":5,"index":"test-index","node":"i-vL8blaT026-wCIWYo16g","reason":{"type":"circuit_breaking_exception","reason":"[request] Data too large, data for [<reused_arrays>] would be [107969413200/100.5gb], which is larger than the limit of [8511422464/7.9gb]","bytes_wanted":107969413200,"bytes_limit":8511422464}},{"shard":6,"index":"test-index","node":"i-vL8blaT026-wCIWYo16g","reason":{"type":"circuit_breaking_exception","reason":"[request] Data too large, data for [<reused_arrays>] would be [106957774000/99.6gb], which is larger than the limit of [8511422464/7.9gb]","bytes_wanted":106957774000,"bytes_limit":8511422464}},{"shard":7,"index":"test-index","node":"i-vL8blaT026-wCIWYo16g","reason":{"type":"circuit_breaking_exception","reason":"[request] Data too large, data for [<reused_arrays>] would be [136670085200/127.2gb], which is larger than the limit of [8511422464/7.9gb]","bytes_wanted":136670085200,"bytes_limit":8511422464}},{"shard":8,"index":"test-index","node":"i-vL8blaT026-wCIWYo16g","reason":{"type":"circuit_breaking_exception","reason":"[request] Data too large, data for [<reused_arrays>] would be [140459030800/130.8gb], which is larger than the limit of [8511422464/7.9gb]","bytes_wanted":140459030800,"bytes_limit":8511422464}}],"caused_by":{"type":"circuit_breaking_exception","reason":"[request] Data too large, data for [<reused_arrays>] would be [116703122800/108.6gb], which is larger than the limit of [8511422464/7.9gb]","bytes_wanted":116703122800,"bytes_limit":8511422464}},"status":503}

> cat t
{
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "*",
            "analyze_wildcard": true
          }
        },
        {
          "range": {
            "ES_timestamp": {
              "gte": 1487183400000,
              "lte": 1487269800000,
              "format": "epoch_millis"
            }
          }
        }
      ],
      "must_not": []
    }
  },
  "size": 0,
  "_source": {
    "excludes": []
  },
  "aggs": {
    "2": {
      "terms": {
        "field": "uuid",
        "size": 500000,
        "order": {
          "1": "desc"
        }
      },
      "aggs": {
        "1": {
          "cardinality": {
            "field": "uuid"
          }
        }
      }
    }
  }
}

This is one full trace of debugging messages: https://gist.github.com/jay-dihenkar/b52d83cdd052b007bdaae409dffe5c60

PS: I have also changed "size": 500000, to 500 as well.

[colings86]

Ok I think the issue here is that the terms aggregation is using the global ordinals execution mode and the ordinal value a collected term is large. The cardinality aggregation pre-allocates memory for all buckets with a ordinal value less than or equal to the current one.

This works well in a lot of cases but in this case the combination of the terms agg using global ordinals and the cardinality agg is making Elasticsearch try to allocate too much memory thus causing the circuit breaker to trip. @jpountz has actually raise an issue about changing this behaviour (#15892).

The good new is that there is an easy workaround for your query that will prove if this is what is happening and get you around it for now. The terms aggregation has an execution_hint field that you can set to ask the terms agg to run in a different mode. Can you set this to global_ordinals_hash and run the same query and let me know what the result is?

So the terms portion of the request should look something like this:

"terms": {
        "field": "uuid",
        "size": 500,
        "execution_hint": "global_ordinals_hash",
        "order": {
          "1": "desc"
        }
      }

[jay-dihenkar]

Hi, This worked! I added it to curl query and ran and it worked.

Is there a way to inject this into Kibana Viz, I already tried to put it in 'advanced' -> json input but it doesn't work saying it's not a recognized option?

[colings86]

@jay-dihenkar glad it worked.

@spalger is the execution_hint on a terms aggregation configurable in Visualize in Kibana?

[jay-dihenkar]

Hi @spalger / @colings86 / @danielmitterdorfer ,

A quick patch in Kibana by some trick or suggestions modifying ./src/core_plugins/console/api_server/es_5_0/aggregations.js to make it work will be appreciated.

[jay-dihenkar]

OK,

Figured out, we can do this from Kibana.
In Edit Viz -> aggs bucket -> advanced -> json -> {"execution_hint": "global_ordinals_hash"}.

@colings86 : Are there any further improvements in the query we can do if we have lots of unique terms to aggregate over other than the execution_hint?

Are there any other significant changes from 2.x to 5.x related to cardinality aggregations and parameters controlling them?