CompositeKeyExtractorTest.testExtractDate fails

[mayya-sharipova]

Test reproducibly fails on elasticsearch 6.3, 6.x, and master on my local Mac laptop:

Last CLI failure on

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+multijob-unix-compatibility/os=fedora/3/console

Reproduce:
./gradlew :x-pack:plugin:sql:test -Dtests.seed=56E62F320F1CC3E3 -Dtests.class=org.elasticsearch.xpack.sql.execution.search.extractor.CompositeKeyExtractorTests -Dtests.method="testExtractDate" -Dtests.security.manager=true -Dtests.locale=he -Dtests.timezone=Atlantic/Faeroe

with an error message:

17:16:06 ERROR   0.06s J2 | CompositeKeyExtractorTests.testExtractDate <<< FAILURES!
17:16:06    > Throwable #1: java.lang.IllegalArgumentException: The datetime zone id 'SystemV/YST9' is not recognised
...
17:16:06    > 	at org.joda.time.DateTimeZone.forTimeZone(DateTimeZone.java:380)
17:16:06    > 	at org.elasticsearch.xpack.sql.execution.search.extractor.CompositeKeyExtractorTests.testExtractDate(CompositeKeyExtractorTests.java:65)

Or another time zone reproduction line:

/gradlew :x-pack:plugin:sql:test -Dtests.seed=A303E5CA896B87E9 -Dtests.class=org.elasticsearch.xpack.sql.execution.search.extractor.CompositeKeyExtractorTests -Dtests.method="testExtractDate" -Dtests.security.manager=true -Dtests.locale=id-ID -Dtests.timezone=Asia/Dhaka

with an error:
Throwable #1: java.lang.IllegalArgumentException: The datetime zone id 'SystemV/AST4' is not recognised

[elasticmachine]

Pinging @elastic/es-search-aggs

[mayya-sharipova]

AwaitsFix was applied for this test on 6.3

[cbuescher]

I'm looking into this. there seems to be some uncovered corner cases when converting from java TimeZone back to joda DateTimeZone.

[cbuescher]

We randomly test with all java.util.TimeZone ids, some of which jodas DateTimeZone.forTimeZone does not recognize. Fortunately those are very odd ones so I guess we can just exclude them from the set of allowed random timezones in the test. I ran the conversion for all java.util.TimeZone ids and those are the ones that don't convert:

The datetime zone id 'SystemV/AST4' is not recognised
The datetime zone id 'SystemV/AST4ADT' is not recognised
The datetime zone id 'SystemV/CST6' is not recognised
The datetime zone id 'SystemV/CST6CDT' is not recognised
The datetime zone id 'SystemV/EST5' is not recognised
The datetime zone id 'SystemV/EST5EDT' is not recognised
The datetime zone id 'SystemV/HST10' is not recognised
The datetime zone id 'SystemV/MST7' is not recognised
The datetime zone id 'SystemV/MST7MDT' is not recognised
The datetime zone id 'SystemV/PST8' is not recognised
The datetime zone id 'SystemV/PST8PDT' is not recognised
The datetime zone id 'SystemV/YST9' is not recognised
The datetime zone id 'SystemV/YST9YDT' is not recognised

I will open a PR that excluded those.

[cbuescher]

For reference: https://github.com/JodaOrg/joda-time/blob/master/src/main/java/org/joda/time/tz/src/systemv

[costin]

This is an old issue that @imotov (currently on holidays) looked at; as @cbuescher is spot on with his assessment.