Terms aggregation shows up irrevelant data

[fvilpoix]

Elasticsearch version (bin/elasticsearch --version): 6.11

Plugins installed: analysis-icu

JVM version (java -version):

openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-1~deb9u1-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

OS version (uname -a if on a Unix-like system):

Linux plop 4.9.0-4-amd64 #1 SMP Debian 4.9.65-3 (2017-12-03) x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

TL;DR Since upgrade to 6.0 (then last 6.1.1), Terms aggregation on integer field shows result on data that should not exists for the provided query.

Original post on forum

Here is a first request I do, in order to assert that I do not have any data > 60:

{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "media_id": "aaa"
          }
        },
        {
          "range": {
            "eng.visu": {
              "gte": 60
            }
          }
        }
      ]
    }
  },
  "size": 9999
} 

eng.visu is an array of 1 to 5 integers, always < 60 for this media_id.

Result is as expected:

{
  "took": 483,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 0,
    "max_score": null,
    "hits": []
  }
}

But then, I do a terms aggregation on those data:

{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "media_id": "aaa"
          }
        }
      ]
    }
  },
  "aggs": {
    "__all__": {
      "terms": {
        "field": "eng.visu",
        "size": 9999
      }
    }
  },
  "size": 0
}

And the result:

{
"took": 24,
"timed_out": false,
"_shards": {
  "total": 5,
  "successful": 5,
  "skipped": 0,
  "failed": 0
},
"hits": {
  "total": 18670,
  "max_score": 0,
  "hits": []
},
"aggregations": {
    "__all__": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": 1,
          "doc_count": 690
        },
        {
          "key": 0,
          "doc_count": 674
        },
        {
          "key": 2,
          "doc_count": 655
        },
        ...
       {
          "key": 80,
          "doc_count": 298
       },
      {
          "key": 82,
          "doc_count": 298
       },
       ...
       {
          "key": 5276,
          "doc_count": 1
        }
      ]
   }
}
}

As you can see, I have keys that are really greater than 60.

[cbuescher]

@fvilpoix thanks for opening this issue, I tried to come up with a reproduction on 6.1.1 with a minimal including a few simple example documents but failed so far. If you are able to reproduce this on a minimal setup (reduced mapping, a small number of examples that exhibit this behaviour on a fresh index) and could share this, that would be ideal. Otherwise, can you provide your example mapping and maybe a few documents from your data so we can reproduce this better?

[fvilpoix]

Thanks for your answer.

I've tried to reduce the dataset to a minimal size in order to reproduce the issue, but as soon as I do a reindexation into a new index, the aggregation works normally.

You can find in attachement the mapping of the migrated index from 5.x, thus some significant data. But the Index as ~30.000.000 documents:

mapping.json.txt
data.json.txt

The data has been extracted with the following query 👍

GET engagement-2017/_search
{
  "aggs": {
    "filtered": {
      "filter": {
        "bool": {
          "filter": [
            {
              "term": {
                "media_id": "f4ab5b1493891cb5"
              }
            },
            {
              "range": {
                "@timestamp": {
                  "gte": "2017-11-11 00:00",
                  "lte": "2017-11-11 01:00",
                  "format": "yyyy-MM-dd HH:mm"
                }
              }
            }
          ]
        }
      },
      "aggs": {
        "__all__": {
          "terms": {
            "field": "eng.visu",
            "size": 9999
          }
        }
      }
    }
  },
  "size": 999
}

You can see that media_id filter is not fullfilled, as many others are fetched. For this media_id, eng.visu should never contains data > 197.

[cbuescher]

Thanks, I don't see anything obviously wrong in the mapping or the query. Just a few things to clarify:

• I assume "engagement-2017" is an alias, since the mapping mentions "engagement_from_prod" as an index name. Can you clarify this and check what indices it points to?
• I see a "_default_", "doc" and "engagement" type in the mapping, are they all still used after migrating to 6.0? If so, is it possible that they accidentally contain any documents with the same media_id with the unexpected "eng.visu" data that might have been removed from other types an now still show up (the aggregation runs across all types I think)

You can see that media_id filter is not fullfilled, as many others are fetched.

I don't completely understand what you mean by "not fullfilled". If you mean that the "hits" in the attached "data.json.txt" contain documents with all kinds of media_id, I think that is because the query part in is a "match_all" and you retrieve 999 hits. Why is this necessary for the aggregation? Maybe I'm missing something.

Other than that, could you also again send the full query (inklusing the request URL) you use to check that for this media id ("f4ab5b1493891cb5") there are no hits with values >197 and if possible the response (maybe redacted to not expose private data)?

[fvilpoix]

Thank you for your help!

• yes, engagement-2017 is an alias, linked on engagement_from_prod. From cerebro:
  
• There are documents into doc and engagement.
  
  Queries/aggregations are done without any type filter, so I believe it matchs all documents from any type, right?

You're right, I've given you the result of the wrong attempt, using an filter aggregation instead of a aggregation with filtered query. Sorry for the wase of time.

I've done the right one, Data is perfect, but aggregation is weird, as there is data > 197 even that there are none in hits!!! (or I haven't understood how aggregations work 😄)

GET engagement-2017/_search?
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "media_id": "f4ab5b1493891cb5"
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "2017-11-11 00:00",
              "lte": "2017-11-11 01:00",
              "format": "yyyy-MM-dd HH:mm"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "__all__": {
      "terms": {
        "field": "eng.visu",
        "size": 9999,
        "order" : { "_key" : "asc" }
      }
    }
  },
  "size": 200
}

data.json.txt

Here is the full query asserting no data exists with values > 197:

GET engagement-2017/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "media_id": "f4ab5b1493891cb5"
          }
        },
        {
          "range": {
            "eng.visu": {
              "gt": 197
            }
          }
        }
      ]
    }
  },
  "size": 9999
} 

And result:

{
  "took": 49,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 0,
    "max_score": null,
    "hits": []
  }
}

[cbuescher]

Many thanks, this is strange indeed. Looking at the aggregation result, there are indeed only 159 documents for "media_id": "f4ab5b1493891cb5" but if I sum up the "doc_counts" in the terms aggregation buckets I get a total of 829, so something with the aggregation scope seems wrong.
@colings86 would you mind sharing your opinion on this when you have time? I might have missed something obvious.

[jimczi]

Thanks @fvilpoix , this is a bug in 6x that appears on indices created in 5x which is why you cannot reproduce when you reindex the data. I am able to reproduce with a fresh install of 6x with an index created in 5x. I opened an issue in Lucene with a patch (https://issues.apache.org/jira/browse/LUCENE-8117) but if we are not able to release a new version of Lucene for the next release (6.2) we'll add a workaround in es directly.

[fvilpoix]

Thanks everyone for the bugfix, I hope it will solve my problem!