Joda time unable to handle some timezones causing test failure

[jaymode]

It appears as though joda time cannot handle certain timezones such as SystemV/MST7MDT, which results in test failures that reproduce locally.

Build link: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/204/console

Reproduce

./gradlew :x-pack:plugin:watcher:test -Dtests.seed=4AA3D2B20D6E347A -Dtests.class=org.elasticsearch.xpack.watcher.trigger.schedule.tool.CronEvalToolTests -Dtests.method="testGetNextValidTimes" -Dtests.security.manager=true -Dtests.locale=hr-HR -Dtests.timezone=SystemV/MST7MDT -Dcompiler.java=11 -Druntime.java=8

Failure

Caused by: java.lang.IllegalArgumentException: The datetime zone id 'SystemV/MST7MDT' is not recognised
	at org.joda.time.DateTimeZone.forTimeZone(DateTimeZone.java:385)
	at org.elasticsearch.xpack.watcher.trigger.schedule.tool.CronEvalTool.<clinit>(CronEvalTool.java:34)

[elasticmachine]

Pinging @elastic/es-core-infra

[rjernst]

The problem is the use of TimeZone.getDefault() in setting the timezone of a "local formatter":

private static final DateTimeFormatter LOCAL_FORMATTER = DateTimeFormat.forPattern("EEE, d MMM yyyy HH:mm:ss Z")
        .withZone(DateTimeZone.forTimeZone(TimeZone.getDefault()));

I wonder if we can remove this local formatter altogether? We go to great lengths to not rely on the local timezone in other parts of the code (which is why randomized runner passes in a random timezone and locale to tests).

[DaveCTurner]

I see the value in supporting non-UTC timezones in this tool. It's for debugging cron expressions, which seem to be evaluated in UTC, and I can imagine one point of confusion is the discrepancy between UTC and the caller's own local timezone. I do however agree it's a pain for testing.

I think another way of looking at the problem is that it's a Joda/java.time mismatch. TimeZone.getDefault() yields a java.util.TimeZone which we then try and convert to an org.joda.time.DateTimeZone but this conversion is lossy. However Joda has its own method for getting the default time zone directly, DateTimeZone.forTimeZone(null), which may fail less. Another option would be to ditch Joda entirely in this tool.

[rjernst]

For now, I've opened #35608 to fix this immediate issue. We will be converting everything to java time separately.

[bizybot]

Just adding it here so we do not miss that the changes are quite recent and were done in #33215. Thanks.