Range query rounding error when used with date_range fields

[deadbeef84]

Elasticsearch version (bin/elasticsearch --version): 6.8.5 and 7.5.0

Plugins installed: default from docker image docker.elastic.co/elasticsearch/elasticsearch:6.8.5 and docker.elastic.co/elasticsearch/elasticsearch:7.5.0

JVM version (java -version):
OpenJDK Runtime Environment AdoptOpenJDK (build 13.0.1+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 13.0.1+9, mixed mode, sharing)

OS version (uname -a if on a Unix-like system):
Linux dd0d4200f531 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 10:36:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

When using date_range type in combination with range query, range lte seems to round down. I would expect this to round up as it does with fields of date type.

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

1. Create index with the following mapping:

curl -X PUT "localhost:9200/mytest?pretty" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "properties": {
      "foo": {
        "type": "date"
      },
      "bar": {
        "type": "date_range"
      }
    }
  }
}
'

2. Index the following document:

curl -X POST "localhost:9200/mytest/_doc/?pretty" -H 'Content-Type: application/json' -d'
{
  "foo": "2019-12-09T12:00:00.000Z",
  "bar": {
    "gte": "2019-12-09T12:00:00.000Z",
    "lte": "2019-12-09T13:00:00.000Z"
  }
}
'

3. Search using range query:

curl -X GET "localhost:9200/mytest/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query":{
    "range":{
      "bar": {
        "gte":"2019-12-09||/d",
        "lte":"2019-12-09||/d"
      }
    }
  }
}
'

Yields no hits. I would expect gte to round to start of day, and lte to round to end of day, but it appears lte rounds to start of day. It works as expected when changing range query to use foo field. It also works correctly if I change lte to 2019-12-10||/d or 2019-12-09T23:59:59.

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[cbuescher]

Hi @deadbeef84, thanks for raising this and the great reproduction.

Looking at the query explain output for the range query on the date_range field gives:

"explanation" : "bar:<ranges:[1575849600000 : 1575849600000]>"

and on the date field we get (after adding another datapoint outside the range so it doesn't rewrite):

 "explanation" : "foo:[1575849600000 TO 1575935999999]"

The different rounding behaviour on the two field types is indeed odd. I wonder if the date_range behaviour has always been like this or if this has recently changed. I tried on 7.5 but will also try a somewhat older 6 version. In any case I think we should discuss changing the behaviour to be similar, although changing might now come with compatibility issues, so even if we decide to change it we might only do it on a mayor version.

[cbuescher]

The date_range field behaviour on 6.8.3 looks the same as described above, query explain is: "explanation" : "bar:<ranges:[1575849600000 : 1575849600000]>"
So even if we consider this a bug or want to change the behaviour we should take that into account. Labling for discussion whether we need to update or clarify in the docs or if we want to change the behaviour to "lte" behaves similar to how if is resolved in date fields.

[cbuescher]

The rounding behaviour for "gt" is also different between range queries on date and on date_range. On the date_range field (bar) the following shows a lower bound of 1575849600001 both for gt and gte.

POST /mytest/_validate/query?explain
{
  "query":{
    "range":{
      "bar": {
        "gt":"2019-12-09||/d"
      }
    }
  }
}

On the date field however, gt rounds up to 1575936000000 while gte includes the lower bound 1575849600000. The behaviour on date is consistent with what we document here: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html#range-query-date-math-rounding while the behaviour on date_range is not. I think we need to change the rounding in RangeType.DATE if we want this to be consistent.

[deadbeef84]

Thanks for digging into this.

I can add that the issue exists on 6.4.1 (which was where we saw it first) and I just tested it on 5.3.0 where behavior is the same. I think date_range type was introduced in 5.2.0 so it doesn't appear to be a regression.

[cbuescher]

Another related observation regarding query_string query: with the above field mapping (foo -> date, bar -> date_range) the following:

POST /mytest/_validate/query?explain
{
  "query": {"query_string": {
    "query": "foo:[2019-12-14 TO 2019-12-14]"
  }}
}

shows the query expands to this query: "explanation" : "foo:[1576281600000 TO 1576367999999]", so it includes the full day "2019-12-14" while the same done on the "date_range" returns "bar:<ranges:[1576281600000 : 1576281600000]>", so to/from are parsed to the same value.

[cbuescher]

Brought this up for discussion and the current tendency is to try and make this consitent, but probably only on 8.0 so we don't change the behaviour on a minor.