[ci] SmokeTestWatcherTestSuiteIT.testMonitorClusterHealth

[andyb-elastic]

Doesn't reproduce, looks like the test cluster had a problem but there's nothing too interesting in the logs. This test has failed a handful of times in the last three months but for different resaons. Not sure there's anything to do here

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=centos/1199/console

CI logs
build-1199-watcher.txt

Integ cluster logs
run.log

REPRODUCE WITH: ./gradlew :x-pack:qa:smoke-test-watcher:integTestRunner \
  -Dtests.seed=D4D1EB8646558B01 \
  -Dtests.class=org.elasticsearch.smoketest.SmokeTestWatcherTestSuiteIT \
  -Dtests.method="testMonitorClusterHealth" \
  -Dtests.security.manager=true \
  -Dtests.locale=ar-LB \
  -Dtests.timezone=America/Cuiaba

[elasticmachine]

Pinging @elastic/es-core-infra

[andyb-elastic]

Actually this does look odd from the integ cluster logs, though it happens after the request failing that fails the test

[2018-07-23T22:10:15,863][ERROR][o.e.s.m.MustacheScriptEngine] Error running com.github.mustachejava.codes.DefaultMustache@56a43cff
com.github.mustachejava.MustacheException: Failed to get value for ctx @[query-template:1]
        at com.github.mustachejava.codes.ValueCode.execute(ValueCode.java:74) ~[compiler-0.9.3.jar:?]
        at com.github.mustachejava.codes.DefaultMustache.run(DefaultMustache.java:34) ~[compiler-0.9.3.jar:?]
        at com.github.mustachejava.codes.DefaultCode.execute(DefaultCode.java:162) ~[compiler-0.9.3.jar:?]
        at com.github.mustachejava.codes.DefaultMustache.execute(DefaultMustache.java:55) ~[compiler-0.9.3.jar:?]
        at com.github.mustachejava.Mustache.execute(Mustache.java:38) ~[compiler-0.9.3.jar:?]
        at org.elasticsearch.script.mustache.MustacheScriptEngine$MustacheExecutableScript.lambda$execute$0(MustacheScriptEngine.java:111) ~[lang-mustache-6.4.0-SNAPSHOT.jar:6.4.0-SNAPSHOT]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at org.elasticsearch.script.mustache.MustacheScriptEngine$MustacheExecutableScript.execute(MustacheScriptEngine.java:110) [lang-mustache-6.4.0-SNAPSHOT.jar:6.4.0-SNAPSHOT]
        at org.elasticsearch.xpack.watcher.common.text.TextTemplateEngine.render(TextTemplateEngine.java:57) [x-pack-watcher-6.4.0-SNAPSHOT.jar:6.4.0-SNAPSHOT]
        at org.elasticsearch.xpack.watcher.actions.logging.ExecutableLoggingAction.execute(ExecutableLoggingAction.java:46) [x-pack-watcher-6.4.0-SNAPSHOT.jar:6.4.0-SNAPSHOT]
        at org.elasticsearch.xpack.core.watcher.actions.ActionWrapper.execute(ActionWrapper.java:144) [x-pack-core-6.4.0-SNAPSHOT.jar:6.4.0-SNAPSHOT] 
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:464) [x-pack-watcher-6.4.0-SNAPSHOT.jar:6.4.0-SNAPSHOT]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:295) [x-pack-watcher-6.4.0-SNAPSHOT.jar:6.4.0-SNAPSHOT]
        at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction$1.doRun(TransportExecuteWatchAction.java:154) [x-pack-watcher-6.4.0-SNAPSHOT.jar:6.4.0-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.4.0-SNAPSHOT.jar:6.4.0-SNAPSHOT]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.4.0-SNAPSHOT.jar:6.4.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:844) [?:?]
Caused by: java.lang.IllegalArgumentException: Iterable object is self-referencing itself
        at org.elasticsearch.common.util.CollectionUtils.ensureNoSelfReferences(CollectionUtils.java:253) ~[elasticsearch-6.4.0-SNAPSHOT.jar:6.4.0-SNAPSHOT]
        at org.elasticsearch.common.util.CollectionUtils.ensureNoSelfReferences(CollectionUtils.java:256) ~[elasticsearch-6.4.0-SNAPSHOT.jar:6.4.0-SNAPSHOT]
        at org.elasticsearch.common.util.CollectionUtils.ensureNoSelfReferences(CollectionUtils.java:256) ~[elasticsearch-6.4.0-SNAPSHOT.jar:6.4.0-SNAPSHOT]
        at org.elasticsearch.common.util.CollectionUtils.ensureNoSelfReferences(CollectionUtils.java:256) ~[elasticsearch-6.4.0-SNAPSHOT.jar:6.4.0-SNAPSHOT]
        at org.elasticsearch.common.util.CollectionUtils.ensureNoSelfReferences(CollectionUtils.java:231) ~[elasticsearch-6.4.0-SNAPSHOT.jar:6.4.0-SNAPSHOT]
        at org.elasticsearch.script.mustache.CustomReflectionObjectHandler.stringify(CustomReflectionObjectHandler.java:160) ~[?:?]
        at com.github.mustachejava.codes.ValueCode.execute(ValueCode.java:69) ~[?:?]
        ... 20 more

[jasontedor]

Good catch @andyb-elastic.

[dnhatn]

Another instance: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/2645/consoleText

[tvernum]

I have enabled x-pack/qa/smoke-test-watcher again.
I have covered the issues that I know about, but there's a comment above

This test has failed a handful of times in the last three months but for different resaons.

I don't have logs for all those, so it's possible that some still exist.
Please re-mute this test if it turns out to still be flaky.

[benwtrent]

Another failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/2183/console

./gradlew :x-pack:qa:smoke-test-watcher:integTestRunner -Dtests.seed=921232DB4961CB4E -Dtests.class=org.elasticsearch.smoketest.SmokeTestWatcherTestSuiteIT -Dtests.method="testMonitorClusterHealth" -Dtests.security.manager=true -Dtests.locale=en-NZ -Dtests.timezone=Europe/Warsaw -Dcompiler.java=11 -Druntime.java=8

12:06:36 FAILURE 2.58s | SmokeTestWatcherTestSuiteIT.testMonitorClusterHealth <<< FAILURES!
12:06:36    > Throwable #1: java.lang.AssertionError: Watch count (from _watcher/stats)
12:06:36    > Expected: is <0>
12:06:36    >      but: was <1>
12:06:36    > 	at __randomizedtesting.SeedInfo.seed([921232DB4961CB4E:61CAABBBBA00A4C4]:0)
12:06:36    > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
12:06:36    > 	at org.elasticsearch.smoketest.SmokeTestWatcherTestSuiteIT.assertWatchCount(SmokeTestWatcherTestSuiteIT.java:230)
12:06:36    > 	at org.elasticsearch.smoketest.SmokeTestWatcherTestSuiteIT.testMonitorClusterHealth(SmokeTestWatcherTestSuiteIT.java:162)
12:06:36    > 	at java.lang.Thread.run(Thread.java:748)
12:06:36 Completed [3/3] in 2.61s, 1 test, 1 failure <<< FAILURES!

[tvernum]

Thanks, I'll look today and re-mute if needed.