[CI] Occasional failures WatchStatusIntegrationTests#testThatStatusGetsUpdated

[cbuescher]

I got this failure locally on master, it reproduces after a few iterations with:

./gradlew :x-pack:plugin:watcher:unitTest -Dtests.seed=A92DF59B55106F76 -Dtests.class=org.elasticsearch.xpack.watcher.watch.WatchStatusIntegrationTests -Dtests.method="testThatStatusGetsUpdated" -Dtests.security.manager=true -Dtests.locale=en-BW -Dtests.timezone=SystemV/YST9 -Dcompiler.java=11 -Druntime.java=11

The error is:

FAILURE 4.67s | WatchStatusIntegrationTests.testThatStatusGetsUpdated <<< FAILURES!
   > Throwable #1: java.lang.AssertionError:
   > Expected: is "2019-02-08T12:25:25.22141Z"
   >      but: was "2019-02-08T12:25:25.221410Z"
   >    at __randomizedtesting.SeedInfo.seed([A92DF59B55106F76:E51670CD660E2260]:0)
   >    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >    at org.elasticsearch.xpack.watcher.watch.WatchStatusIntegrationTests.testThatStatusGetsUpdated(WatchStatusIntegrationTests.java:49)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   >    at java.base/java.lang.Thread.run(Thread.java:834)

This looks like another of the cases where the new java time formatting swallows trailing zeros. @pgomulka could you take a look, I remember reviewing #38505 which looked similar.

[elasticmachine]

Pinging @elastic/es-core-features

[cbuescher]

Muted on master, 7.x (779673c) and 7.0 (cfde3de)

[pgomulka]

interestingly WatchStatus.lastChecked or WatchStatus.lastMetCondition are using a different formatting then WatchStatus.state.timestamp when converting toXcontent

The lastChecked and lastMetCondition are formatted using XContentElasticsearchExtensions

        transformers.put(ZonedDateTime.class, d -> DEFAULT_FORMATTER.format((ZonedDateTime) d));

where DEFAULT_FORMATTER is strict_date_optional_time_nanos

Whereas WatchStatus.state.timestamp is using:

WatcherDateTimeUtils.writeDate(Field.TIMESTAMP.getPreferredName(), builder, timestamp)

with "strict_date_optional_time||epoch_millis"

These are different for instance when formatting 2019-02-08T13:37:06.**687670Z**
WatcherDateTimeUtils.dateTimeFormatter -> 2019-02-08T13:37:06.687Z
XContentElasticsearchExtension.DEFAULT_FORMATTER -> 2019-02-08T13:37:06.68767Z

@cbuescher @spinscale Do you think this is intentional?

[spinscale]

on top of my head I'd say we keep all on milliseconds, there is no need for any nanosecond granularity in watcher, as the date field is only in millisecond resolution anyway.