Core: Migrating from joda time to java.time

[spinscale]

Objective: In order to support nanosecond resolution and to remove the joda time dependency from Elasticsearch, we need to migrate a lot of code over to java.time.

This is a meta issue detailing the steps that need to be taken to migrate away from joda time and to java.time.

There are a few things to keep in mind and discuss before outlining the single steps

Retaining our gazillion date formats

Do we want to retain the tons of our named date formats, named very well like strictOrdinalDateTimeNoMillis or strictWeekDateTimeNoMillis (also we support snake and camel case for each of those!)? I think we could built a layer that is still able to read those, but then maybe convert it to formatted dates like xxxx-'W'ww-e'T'HH:mm:ssZZ (just an example). This way we could get rid of this mapping entirely or at least just maintain it for a few examples like epoch_millis, epoch_second and dateOptionalTime as the text representation are nowhere near as readable as the date time formats, once internalized.

You can find the full list in the Joda class

BWC breaking

Some classes used by builders to create queries are using joda time classes. As we move towards the REST client and away from the Transport Client this will be less of an BWC issue.

Steps

• Ensure time zones are parsed the same way in joda and java.time, other wise the timezone parameters i.e. from the query string query/aggregations could act differently
• Make DateMathParser use java.time
• Make IndexNameExpressionResolver.DateMathExpressionResolver use java.time, also remove the check for the date_math_expression_resolver.default_time_zone and date_math_expression_resolver.default_date_format settings here (they are not registered anyway). Core: Move IndexNameExpressionResolver to java time #34507
• Ensure new java time IndexNameExpressionResolver is BWC compatible when formatters are used
• Convert field mapper types to use ZonedDateTime Core: Switch mapping/aggregations over to java time #36363
• Convert DateTimeUnit, Rounding, TimeValue classes
• Convert XContentBuilder, StreamInput, StreamOutput classes (the serialized version of this is just a long and a timezone in our streams, so those are easy to convert. Same applies for the XContentBuilder)
• Convert the cat API classes (super easy, mostly just formatters, that get their input as milliseconds)
• SortingNumericDocValues and classes extending from it in the fielddata package need to be converted (or removed?) Core: Switch mapping/aggregations over to java time #36363
• Replace joda with java.time in painless
• Replace joda with java.time in ingest Replace joda time in ingest-common module #38088
• Mapping: Replace in DocValueFormat and DateFieldMapper Core: Switch mapping/aggregations over to java time #36363
• Remove jodatime dependency
• Add support to nanosecond resolution, add new date format time that supports nanoseconds and ensure that the formatter supports it properly Core: Add new nanosecond supporting field mapper #32601
• Remove usages of joda in x-pack/ML package Core: Switch mapping/aggregations over to java time #36363 (partially)
• Remove usages of joda in Monitoring Core: Migrating from joda to java.time. Monitoring plugin #36297
• Remove usages of joda in Watcher Migrating from joda to java.time. Watcher plugin  #35809
• Refactor DateHistograms bucket aggregations to use java.time Core: Switch mapping/aggregations over to java time #36363

[clintongormley]

This way we could get rid of this mapping entirely or at least just maintain it for a few examples like epoch_millis, epoch_second and dateOptionalTime as the text representation are nowhere near as readable as the date time formats, once internalized.

++

We can deprecate their use in 6.x and say ("please replace with YYYY..."), then on upgrade to 7.0, we can auto-upgrade the mappings to use the equivalent date pattern format.

[spinscale]

some more discussion about BWC.

Painless

Right now calling doc['foo'].value returns a joda DateTime object (due to ScriptDocValues in core returning one), which makes it non-trivial to just convert to java time, as many users will have put this kind of code in their scripts to deal with dates.

A possible work around for this would be to have a doc['foo'].date_value object which returns a java time.

Client side runtime casting of bucket keys

InternalDateHistogram.Bucket.getKey() implements the Bucket interface, which returns Object for the getKey() method. In the case of the date histogram a joda DateTime is returned, which clients might just be casting to DateTime object, because it has always been like that. This might only be uncovered in runtime, and not in compile time for people using Java.

We need to check if the high level REST client is affected by this as well.

Different symbols used in date formatting

There are a couple of differences when date time formatters are used, according to the java docs. joda time has a century of era symbol, using the C character. This has been removed fully from java time. Also the YYYY formatter is year of era in joda time, but week based year in java time (need to check how much of an impact this really is).

One idea was to have a setting which allows to set which date implementation would be used, but this would mean to not have support rolling upgrades as having to have a hard move from one implementation to the other, so we need to support both time libraries at least in scripting.

[spinscale]

There are some differences in TZ handling we have to cater for. Jodatime has 593 tz ids, java time 600. java time adds a few new ones (starting with SystemV which seem not to be used at all), and those can be ignored. However, java time removes EST, GMT+0, GMT-0, HST, MST, ROC.

The general recommendation seems to be to use either offsets, like UTC-08:00 or those region based names like America/New York rather than the above as those are ambiguous.

We might need to add deprecation warnings for the above timezones.