Histogram bucketing negative values incorrectly

[baelter]

I've been trying to make sense of Elasticssearch histogram aggregations the last couple of days. And I've found that they don't work as expected, or even advertised.

Lets say i want to aggregate like so:

"aggregations": {
  "sea_water_temperature": {
    "histogram": {
      "field": "sea_water_temperature",
      "interval": 3
    }
  }
}

Response buckets looks fine at first glance, but when trying to query for documents within the bounds of a bucket I don't get the same document count as the bucket suggested. E.g.

"filter": {
  "range": {
    "sea_water_temperature": {
      "lt": 0,
      "gte": -3
    }
  }
}

This could give x results while the bucket "-3" had a doc_count of y. This seems to only be an issue for negative bucket keys.

In the docs for histogram it states that the bucket key for a given value is:

rem = value % interval
if (rem < 0) {
  rem += interval
}
bucket_key = value - rem

However I tried a term aggregation with that as a value script:

"aggregations": {
  "sea_water_temperature": {
    "terms": {
      "field": "sea_water_temperature",
      "script": "rem = _value % interval; rem = rem < 0 ? rem + interval : rem; _value - rem",
      "params": {
        "interval": 3
      }
    }
  }
}

That gives me the same kind of bucketing as histogram does but now my filter queries actually match the doc_counts of the buckets(!). Why isn't histogram working as described? or am I missing something?

[clintongormley]

I can replicate this:

DELETE /_all 

POST /t/t/_bulk
{ "create": {}}
{  "t": -3.9}
{ "create": {}}
{  "t": -2.6}
{ "create": {}}
{  "t": -1.4}
{ "create": {}}
{  "t": -0.2}
{ "create": {}}
{  "t": 0.2}
{ "create": {}}
{  "t": 2.4}
{ "create": {}}
{  "t": 2.8}
{ "create": {}}
{  "t": 3.9}

GET /_search?search_type=count
{
  "aggs": {
    "NAME": {
      "histogram": {
        "field": "t",
        "interval": 3
      }
    }
  }
}

Returns:

"aggregations": {
  "NAME": {
     "buckets": [
        {
           "key": -3,
           "doc_count": 3
        },
        {
           "key": 0,
           "doc_count": 4
        },
        {
           "key": 3,
           "doc_count": 1
        }
     ]
  }
}

[baelter]

Can I contribute in any way?

[clintongormley]

@baelter The problem here is that the histogram casts floats to integers, which is why some negative numbers are being added to the wrong bucket. @jpountz has a solution here, he just needs to implement it.

[jpountz]

It is a general limitation of histograms today that they do neither work on decimal values (because of negative values, as you noticed) nor on decimal intervals (intervals are required to be an integer). I agree that it would be nice to fix both issues.

[baelter]

It would, in the mean time, a term aggregation with histogram key function as a script will do as a temp fix.

[jpountz]

For reference here is the related (though not duplicate) issue #4847

[sylvinus]

I would strongly recommend mentioning this in the docs until it is fixed. It's a subtly broken behavior and it would prevent more users losing time (or their minds! :)

[clintongormley]

@sylvinus agreed - feel like sending a docs PR?