date_histogram fails for year buckets >1, like "10y"

[jordansissel]

Attempting to do a date_histogram with 10-year sized buckets, passing "10y" to the interval setting results in this error:

ElasticsearchParseException[Failed to parse [10y]]; nested: NumberFormatException[For input string: \"10y\"]; 

As a workaround, I can specify large values for "weeks", so swapping in 520w (or 3650d) might be inaccurate, but possibly acceptable to some folks.

Sample agg (taken from Kibana 4) that causes the NumberFormatException:

          "date_histogram": {
            "field": "@timestamp",
            "interval": "1y",
            "min_doc_count": 1,
            "extended_bounds": {
              "min": -1767196800000,
              "max": 1418425008298
            }
          }

[rjernst]

It looks like TimeValue.parseTimeValue() is missing support for year:
https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/unit/TimeValue.java#L228

[jordansissel]

Any thoughts on whether the multiplier should be 365 days or 365.25?

[rjernst]

That's a tough question. Year is sort of an imprecise time measurement because the length changes. 1y and year are handled completely separately (as a "unit" instead of an "interval") in TimeZoneRounding. It looks like the underlying ImpreciseDateField for a year uses the average length of a year, which from a quick test is 31556952000 milliseconds (1.0000008239079 years according to google).

[jordansissel]

I don't see any tests for parseTimeValue, so I can add those; rough proposal for this work being:

1. add "y" support such that things like "2y" work
2. add tests for all cases of parseTimeValue
3. figure out the multiplier unit for 'year' length

[rjernst]

To be precise, I think we need to think about what someone means if they do something like your "2y" example. By converting an imprecise time to milliseconds, I think we would have inaccuracies around the endpoints? For example, if a doc had a date that was the first second of a year, it could be counted in the previous year. It is an extreme case, but possible, and the results would be confusing.

I think when someone says 2y, what they mean is to look at just the year portion of the date, and bucket in increments of 2 there. So it seems like the current "rounding" (what is used to extract the key for a value in the histogram) needs to be rethought a bit to properly support year (and units above "hour").

[jordansissel]

yeah I think "year" being a duragion (365 days) not necessarily january 1 to dec 31.

Kibana has a way to compensate by saying "Round down" and it puts the boundaries at January 1, etc.

[jordansissel]

@rjernst I think the problem you're describing is the difference really between a "calendar week" and a "week-long duration". Calendar week/year/month start on known boundaries w/ known length, where as a duration has unknown boundary but known length.

[dadoonet]

Linking to #2473 issue.
I never found a way for implementing it properly. Switching to ms won't produce expected results.

[colings86]

This issue relates to #7666.

They are different bits of functionality but both suffer from the same problem. I tend to agree with @rjernst and @dadoonet in terms of the user's expectations. The difference with weeks is that the calendar week and "week-long duration" are both always the same length (the difference being that the calendar week has a definite start) whereas for months and years both the calendar and duration variety vary in length. As a user I would always expect a month duration starting on the 5th would end on the 5th (exclusive) the next month.

A PR was submitted for the above issue (#7796) but we would have to look into whether the same approach is appropriate/possible here.

[rkalyankumar]

for more than 1 year, let's say 2 year interval the interval should be specified as 730d (365 * 2) & it works well. I have tested this until 3650d (10 year intervals) & works good.

[clintongormley]

Related to #14802