Date field does not support partial dates anymore

[ywelsch]

On ES 6.x and below, date fields (i.e. fields of "type": "date") used to accept values of the form "2018-09" (i.e. just year and month) whereas this no longer works on ES 7.x (tested on 7.3.0 and master).

This breaks recoveries from 6.x to 7.x and indexing pipelines.

[elasticmachine]

Pinging @elastic/es-core-infra

[rjernst]

The default parser for date fields is strict_date_optional_time (and a fallback to epoch millis, but that is unimportant for this discussion). It looks like confusing terminology in Joda has caused us to be too strict in several places with our conversion to java time. It looks like myself and others who worked on the java time migration viewed the "strict" term here as correlated with the "optional" portion, meaning the above format would have a required date, but an optional time. But upon further inspection it appears strict has to do with validation of the fields of the date, eg allowing 13 as the month. Unfortunately the "dueling" tests that we have comparing java time to joda did not compare these partial dates to joda.

Java time's facility for allowing lenient parsing is the ResolverStyle, which can be LENIENT, SMART or STRICT. SMART is actually the default, which will try to carry over field values out of the expected range (for example, july 32 would be translated to aug 1). Switching the strict parsers we have to allow optional portions of dates, and changing the resolver style to strict should match the old joda behavior.

We should also be consistent about not using SMART, and maybe discuss whether we need the LENIENT resolved versions of date formatters at all? It is unclear how these would be used in Elasticsearch since in many places we necessarily convert date/times to epoch millis or nanos, and I don't think a lenient parsed field can be converted in this way without a custom field resolver.

/cc @spinscale @pgomulka

[pgomulka]

So we can fix 4 things here:

1. To enable partial parsing of dates. So that date_optional_time can parse 2018, 2018-01, 2018-01-01
2. To make the strict_* parsers validate the length of date parts. This would make date_optional_time parse 99999-01-01 but strict_date_optional_time would fail. This is the difference between StrictISODateTimeFormat and ISODateTimeFormat in Joda implementation. I would however skip this one, as we already relaxed this in ES v7, some people might have started to use it.
3. Enable ResolverStyle.Strict for all parsers. This is how it worked in Joda (there was no option to change this). By default Java is using ResolverStyle.SMART which would allow parsing 2018-02-30 but Joda would throw an exception.
4. Consider renaming our formatters. strict_date_optional_time makes it unclear what is strict here (in fact it is the strict length parsing, not the presence of full date part or resolver style - which is always strict). Maybe we could start with adding aliases for these unclear. Like strict_length_date_time ? Or any other ideas?