Date range query rewrite should rewrite INTERSECTS query case in UTC

[colings86]

The rewrite for the date range query has three cases:

1. DISJOINT - query is rewritten to a match none query
2. WITHIN - query is rewritten as an unbounded ([null TO null]) date range query
3. INTERSECTS - query is not rewritten and this is returned

Because we don't rewrite the query in the 3rd case above we miss the opportunity to have the query returned in a normalised (UTC milliseconds) form. This means the following cases (and possibly others I haven't thought of) will result in cache misses when the queries are actually the same:

• Two users submit queries in different timezones which map to the same UTC time range. For example, user A searches for [9pm TO 10pm] GMT and user B searches for [10pm TO 11pm] CET, both queries actually map to [9pm TO 10pm] UTC when run on the shard.
• Two users search for the same time range but with different valid date formats (because we allow multiple date formats to be used). For example, user A searches for [2012-01-01 TO 2014-01-01] and user B searches for [01/01/2012 TO 01/01/2014].
• One user searches for a date math expression using now that maps directly to another users exact search. For example, user A searches for [2016-01-01 TO 2017-01-01] and user B searches for [now-1y/y TO now/y]

In all of these cases we can use the same solution; in the INTERSECTS case we can rewrite the query so the timezone is UTC and the from and to values are in epoch milliseconds. Then the query we be more likely to cause a hit on the cache.

[s1monw]

sounds like this makes sense

[jpountz]

I agree this would be nice, but I suspect it might be tricky to do in practice since not all date fields would accept UTC millis. Or we would need to add a mechanism to tell date fields to not try to parse the incoming value but just interpret it as a number of millis since UTC. I have the feeling we already have too many if (fieldType instanceof DateFieldType) so I am not sure I would be happy to pay this price. Or maybe there are cleaner ways to do this that I have not thought about?

[colings86]

since not all date fields would accept UTC millis

I thought if you gave the from and to as a Long instead of String it would interpret it as epoch millis? I thought that the date format only comes into play if from and/or to are String/BytesRef? Or maybe that is not what you mean?

[jpountz]

I just looked at the code and apparently we always parse the toStringed representation of the date. Which I think is the right thing to do, otherwise it would be very confusing when the format is epoch_second that strings get parsed as seconds but longs as millis?

[colings86]

Ok, my mistake. I think you are right that it would be confusing if the format was epoch_seconds and by a long value was interpreted as epoch_millis. Maybe we could use the format in the RangeQueryBuilder so that when we create the rewritten query it has a UTC timezone a from and to that are in epoch_millis and a format of epoch_millis?

[jpountz]

I think that would work.

[colings86]

Looking at this again I see that even with my above suggestion we will need to add instanceof checks ands special case date and range fields. Given the complexity this will add I don't think this complexity will be outweighed by the benefit of better caching. This is especially because this issue was born from theory and not from seeing this happening in real life.

For those reasons I am going to close this issue for now, if people find that the lack of cache hits between queries with different timezones or date formats is having a measurable impact on their query performance we can reopen and discuss this issue again.