Using IP in date histogram aggregate causes array_index_out_of_bounds_exception

[angorayc]

Elasticsearch version (bin/elasticsearch --version): 7.12

Plugins installed: SecuritySolution

Description of the problem including expected versus actual behavior:

Steps to reproduce:

1. I ran this query, and it returns array_index_out_of_bounds_exception often:

GET .siem-signals-siem-estc-dev-default,auditbeat-*,filebeat-*,logs-*,packetbeat-*,winlogbeat-*,-*elastic-cloud-logs-*/_search
{
    "aggregations": {
      "eventActionGroup": {
        "terms": {
          "field": "destination.ip",
          "missing": "0.0.0.0",
          "order": {
            "_count": "desc"
          },
          "size": 10
        },
        "aggs": {
          "events": {
            "date_histogram": {
              "field": "@timestamp",
              "fixed_interval": "2699999ms",
              "min_doc_count": 0,
              "extended_bounds": {
                "min": 1618959600000,
                "max": 1619045999999
              }
            }
          }
        }
      }
    },
    "query": {
      "bool": {
        "filter": [
          {
            "bool": {
              "must": [],
              "filter": [
                {
                  "match_all": {}
                }
              ],
              "should": [],
              "must_not": []
            }
          },
          {
            "exists": {
              "field": "destination.ip"
            }
          },
          {
            "range": {
              "@timestamp": {
                "gte": "2021-04-20T23:00:00.000Z",
                "lte": "2021-04-21T22:59:59.999Z",
                "format": "strict_date_optional_time"
              }
            }
          }
        ]
      }
    },
    "size": 0
  }

2. It returns

{
  "error" : {
    "root_cause" : [
      {
        "type" : "array_index_out_of_bounds_exception",
        "reason" : "Index 16 out of bounds for length 16"
      }
    ],
    "type" : "array_index_out_of_bounds_exception",
    "reason" : "Index 16 out of bounds for length 16"
  },
  "status" : 500
}

Provide logs (if relevant):

I enabled error_trace=true, and here is what I've got:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "array_index_out_of_bounds_exception",
        "reason" : "Index 16 out of bounds for length 16",
        "stack_trace" : "org.elasticsearch.ElasticsearchException$1: Index 16 out of bounds for length 16\n\tat org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:633)\n\tat org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:561)\n\tat org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:138)\n\tat org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:99)\n\tat org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:82)\n\tat org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:47)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:38)\n\tat org.elasticsearch.rest.action.RestCancellableNodeClient$1.onResponse(RestCancellableNodeClient.java:87)\n\tat org.elasticsearch.rest.action.RestCancellableNodeClient$1.onResponse(RestCancellableNodeClient.java:81)\n\tat org.elasticsearch.client.node.NodeClient.lambda$executeLocally$0(NodeClient.java:100)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:170)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:164)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:163)\n\tat org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:214)\n\tat org.elasticsearch.action.ActionListener$RunAfterActionListener.onResponse(ActionListener.java:310)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:630)\n\tat org.elasticsearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:109)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:397)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:391)\n\tat org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:219)\n\tat org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:101)\n\tat org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:117)\n\tat org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:84)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:728)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\nCaused by: java.lang.ArrayIndexOutOfBoundsException: Index 16 out of bounds for length 16\n\tat org.apache.lucene.util.UnicodeUtil.UTF8toUTF16(UnicodeUtil.java:602)\n\tat org.apache.lucene.util.BytesRef.utf8ToString(BytesRef.java:137)\n\tat org.elasticsearch.search.DocValueFormat$1.format(DocValueFormat.java:106)\n\tat org.elasticsearch.search.DocValueFormat$1.format(DocValueFormat.java:83)\n\tat org.elasticsearch.search.aggregations.bucket.terms.StringTerms$Bucket.getKeyAsString(StringTerms.java:71)\n\tat org.elasticsearch.search.aggregations.bucket.terms.StringTerms$Bucket.keyToXContent(StringTerms.java:81)\n\tat org.elasticsearch.search.aggregations.bucket.terms.InternalTerms$Bucket.toXContent(InternalTerms.java:123)\n\tat org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.doXContentCommon(AbstractInternalTerms.java:342)\n\tat org.elasticsearch.search.aggregations.bucket.terms.InternalMappedTerms.doXContentBody(InternalMappedTerms.java:127)\n\tat org.elasticsearch.search.aggregations.InternalAggregation.toXContent(InternalAggregation.java:288)\n\tat org.elasticsearch.search.aggregations.Aggregations.toXContentInternal(Aggregations.java:119)\n\tat org.elasticsearch.search.aggregations.Aggregations.toXContent(Aggregations.java:110)\n\tat org.elasticsearch.action.search.SearchResponseSections.toXContent(SearchResponseSections.java:97)\n\tat org.elasticsearch.action.search.SearchResponse.innerToXContent(SearchResponse.java:267)\n\tat org.elasticsearch.action.search.SearchResponse.toXContent(SearchResponse.java:244)\n\tat org.elasticsearch.rest.action.RestStatusToXContentListener.buildResponse(RestStatusToXContentListener.java:46)\n\tat org.elasticsearch.rest.action.RestStatusToXContentListener.buildResponse(RestStatusToXContentListener.java:22)\n\tat org.elasticsearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:31)\n\tat org.elasticsearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:23)\n\tat org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:26)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:36)\n\t... 24 more\n"
      }
    ],
    "type" : "array_index_out_of_bounds_exception",
    "reason" : "Index 16 out of bounds for length 16",
    "stack_trace" : "java.lang.ArrayIndexOutOfBoundsException: Index 16 out of bounds for length 16\n\tat org.apache.lucene.util.UnicodeUtil.UTF8toUTF16(UnicodeUtil.java:602)\n\tat org.apache.lucene.util.BytesRef.utf8ToString(BytesRef.java:137)\n\tat org.elasticsearch.search.DocValueFormat$1.format(DocValueFormat.java:106)\n\tat org.elasticsearch.search.DocValueFormat$1.format(DocValueFormat.java:83)\n\tat org.elasticsearch.search.aggregations.bucket.terms.StringTerms$Bucket.getKeyAsString(StringTerms.java:71)\n\tat org.elasticsearch.search.aggregations.bucket.terms.StringTerms$Bucket.keyToXContent(StringTerms.java:81)\n\tat org.elasticsearch.search.aggregations.bucket.terms.InternalTerms$Bucket.toXContent(InternalTerms.java:123)\n\tat org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.doXContentCommon(AbstractInternalTerms.java:342)\n\tat org.elasticsearch.search.aggregations.bucket.terms.InternalMappedTerms.doXContentBody(InternalMappedTerms.java:127)\n\tat org.elasticsearch.search.aggregations.InternalAggregation.toXContent(InternalAggregation.java:288)\n\tat org.elasticsearch.search.aggregations.Aggregations.toXContentInternal(Aggregations.java:119)\n\tat org.elasticsearch.search.aggregations.Aggregations.toXContent(Aggregations.java:110)\n\tat org.elasticsearch.action.search.SearchResponseSections.toXContent(SearchResponseSections.java:97)\n\tat org.elasticsearch.action.search.SearchResponse.innerToXContent(SearchResponse.java:267)\n\tat org.elasticsearch.action.search.SearchResponse.toXContent(SearchResponse.java:244)\n\tat org.elasticsearch.rest.action.RestStatusToXContentListener.buildResponse(RestStatusToXContentListener.java:46)\n\tat org.elasticsearch.rest.action.RestStatusToXContentListener.buildResponse(RestStatusToXContentListener.java:22)\n\tat org.elasticsearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:31)\n\tat org.elasticsearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:23)\n\tat org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:26)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:36)\n\tat org.elasticsearch.rest.action.RestCancellableNodeClient$1.onResponse(RestCancellableNodeClient.java:87)\n\tat org.elasticsearch.rest.action.RestCancellableNodeClient$1.onResponse(RestCancellableNodeClient.java:81)\n\tat org.elasticsearch.client.node.NodeClient.lambda$executeLocally$0(NodeClient.java:100)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:170)\n\tat org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:164)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:163)\n\tat org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:214)\n\tat org.elasticsearch.action.ActionListener$RunAfterActionListener.onResponse(ActionListener.java:310)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:630)\n\tat org.elasticsearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:109)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:397)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:391)\n\tat org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:219)\n\tat org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:101)\n\tat org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:117)\n\tat org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:84)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:728)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\n"
  },
  "status" : 500
}

Here is how SecuritySolution consume this query:
elastic/kibana#97812 (comment)

[elasticmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[not-napoleon]

Thanks for reporting this, I'll take a look.

[not-napoleon]

@angorayc would it be possible to get the mappings for the indices this is running against? It looks like it's matching a lot of indices and I'm wondering if there's some mapping inconsistency that's leading to this behavior.

[not-napoleon]

Update: I haven't yet been able to reproduce this directly, but by running the query over two indexes, one of which uses an IP field mapping and one of which uses a keyword field mapping, I've gotten a similar error in the same part of the code. Nothing else I've tried (variations on mixing IPv4, IPv6 and Missing values) has yielded up an error so far. @angorayc sent me the mappings for one of the involved indices and I'm still looking at that (it's really big. I need to brush up on my jq syntax), but I strongly suspect that the destination.ip field is not mapped as an IP in one or more of the indexes being queried. I'm going to see what I can do about providing a better error message in that case, but I'm not sure it's possible for the agg to operate correctly with a bad mapping, so we may not be able to do better than improving the error message.

It's still possible there's some other cause, since I don't have a reliable minimum reproduction case yet. More investigation is needed, I'll keep you posted.

[imotov]

I was hoping I took care of it in #70839 but it looks like a similar issue that still exist in master. Here is a repro:

DELETE test1,test2

PUT test1
{
  "mappings": {

  }  
}

PUT test2
{
  "mappings": {
    "properties": {
      "val": {
        "type": "ip"
      }
    }
  }
}


PUT test1/_doc/1
{
}

PUT test2/_doc/1
{
  "val": "255.255.255.255"
}

POST test2,test1/_search
{
  "aggs": {
    "terms": {
      "terms": {
        "field": "val",
        "missing": "0.0.0.0", 
        "size": 10
      }
    }
  }
}

[FrankHassanabad]

@imotov , @not-napoleon, The above unmapped test case does look like the root of what we have been seeing. We played with estc more, and narrowed it down to unmapped types causing the problems with the missing.

[not-napoleon]

Well, I have good news for you then, you can fix this with a one-liner. Simply add "value_type": "ip" to your terms aggregation, and it should work, like so:

{
  "aggs": {
    "terms": {
      "terms": {
        "field": "val",
        "missing": "0.0.0.0", 
        "size": 10,
        "value_type": "ip"
      }
    }
  }
}

What's happening here is that the aggregation usually gets it's ValuesSourceType (which decides what formatter to use) from the field info. If there's no field info, it has a default to fall back to, which for Terms is 'Keyword'. So in this case, you just need to tell it explicitly to use the IP type, and you should be all set.

I'll see if I can do some work to improve the error message, but hopefully this solves your immediate problem. Let me know if we can do anything else for you.