MonitoringIT testMonitoringService fails in CI

[elasticmachine]

Original comment by @jaymode:

The MonitoringIT#testMonitoringService fails (but not reproducibly) in CI due to not finding documents that the test expects:

07:56:49 FAILURE 10.5s J1 | MonitoringIT.testMonitoringService <<< FAILURES!
07:56:49    > Throwable LINK REDACTED: java.lang.AssertionError: No monitoring documents yet
07:56:49    > Expected: a value greater than <0L>
07:56:49    >      but: <0L> was equal to <0L>
07:56:49    > 	at __randomizedtesting.SeedInfo.seed([7CC06B70BE47318E:572522E2D7107E38]:0)
07:56:49    > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
07:56:49    > 	at org.elasticsearch.xpack.monitoring.integration.MonitoringIT.lambda$enableMonitoring$13(MonitoringIT.java:542)
07:56:49    > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:734)
07:56:49    > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:708)
07:56:49    > 	at org.elasticsearch.xpack.monitoring.integration.MonitoringIT.enableMonitoring(MonitoringIT.java:537)
07:56:49    > 	at org.elasticsearch.xpack.monitoring.integration.MonitoringIT.whenExportersAreReady(MonitoringIT.java:511)
07:56:49    > 	at org.elasticsearch.xpack.monitoring.integration.MonitoringIT.testMonitoringService(MonitoringIT.java:214)
07:56:49    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
07:56:49    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
07:56:49    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
07:56:49    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
07:56:49    > 	at java.base/java.lang.Thread.run(Thread.java:844)
07:56:49    > 	Suppressed: java.lang.AssertionError: No monitoring documents yet

Failure links:
LINK REDACTED
LINK REDACTED
LINK REDACTED
LINK REDACTED

I am muting this test on master and 6.x. Assigning to @pickypg and @tlrx based on git history.

[jpountz]

We discussed this test and agreed to enable it again as it made little sense to use that this one would fail but not testMonitoringBulk.

[nik9000]

This consistently fails for me in 6.x:

 ./gradlew :x-pack:plugin:monitoring:internalClusterTest -Dtests.seed=A3F2FBB54E7775C4 -Dtests.class=org.elasticsearch.xpack.monitoring.integration.MonitoringIT -Dtests.method="testMonitoringService" -Dtests.security.manager=true -Dtests.locale=de -Dtests.timezone=Australia/Tasmania -Dcompiler.java=10 -Druntime.java=8

[nik9000]

I'm fixing this test and will push an update.

[nik9000]

I pushed f311680 . That doesn't fix the failure for which this test was originally created but it does get it passing again. Now we can get back to hunting the rare failure.

[javanna]

New failure, same as the original one: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java8,nodes=virtual&&linux/265/console . I wasn't able to reproduce it.

[jakelandis]

Another failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-windows-compatibility/1852/console

Unable to reproduce on Linux. (I don't think this is a Windows specific failure)

REPRODUCE WITH: gradlew :x-pack:plugin:monitoring:internalClusterTest \
  -Dtests.seed=B61E9FE6C95A99BE \
  -Dtests.class=org.elasticsearch.xpack.monitoring.integration.MonitoringIT \
  -Dtests.method="testMonitoringService" \
  -Dtests.security.manager=true \
  -Dtests.locale=hu \
  -Dtests.timezone=Atlantic/Cape_Verde \
  -Dcompiler.java=10 \
  -Druntime.java=8