[CI] Failure in WatcherStatsTests.testStartedStats

[elasticmachine]

Original comment by @droberts195:

The test failed in this build: LINK REDACTED

The REPRO command is:

gradlew :x-pack-elasticsearch:plugin:watcher:test \
  -Dtests.seed=2CB17D5634142025 \
  -Dtests.class=org.elasticsearch.xpack.watcher.transport.action.stats.WatcherStatsTests \
  -Dtests.method="testStartedStats" \
  -Dtests.security.manager=true \
  -Dtests.locale=es-SV \
  -Dtests.timezone=America/Barbados

This did not reproduce locally for me.

The exception is:

10:52:13    > Throwable LINK REDACTED: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_t0, v2=STOPPED]]
10:52:13    > Expected: every item is is <STARTED>
10:52:13    >      but: an item was <STOPPED>
10:52:13    > 	at __randomizedtesting.SeedInfo.seed([2CB17D5634142025:CE7953EBF56DFD42]:0)
10:52:13    > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
10:52:13    > 	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:505)
10:52:13    > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:731)
10:52:13    > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:705)
10:52:13    > 	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.startWatcher(AbstractWatcherIntegrationTestCase.java:491)
10:52:13    > 	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase._setup(AbstractWatcherIntegrationTestCase.java:210)
10:52:13    > 	at java.lang.Thread.run(Thread.java:748)

[elasticmachine]

Original comment by @droberts195:

There has also been another Watcher test failure today, and although it's a different test the underlying problem is again that Watcher is not started when it is expected to be started.

The failure is in LINK REDACTED

The REPRO command is:

./gradlew :x-pack-elasticsearch:plugin:watcher:test \
  -Dtests.seed=15D5EB831C7590D \
  -Dtests.class=org.elasticsearch.xpack.watcher.history.HistoryActionConditionTests \
  -Dtests.method="testActionConditionWithFailures" \
  -Dtests.security.manager=true \
  -Dtests.locale=ko \
  -Dtests.timezone=Canada/Saskatchewan

Once again it does not reproduce locally for me.

The exception that causes the test failure is this:

11:30:40    > Throwable LINK REDACTED: java.lang.AssertionError: could not find executed watch record
11:30:40    > Expected: a value equal to or greater than <1L>
11:30:40    >      but: <0L> was less than <1L>
11:30:40    > 	at __randomizedtesting.SeedInfo.seed([15D5EB831C7590D:B2EAA6FAA24C20B5]:0)
11:30:40    > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
11:30:40    > 	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$assertWatchWithMinimumActionsCount$2(AbstractWatcherIntegrationTestCase.java:472)
11:30:40    > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:731)
11:30:40    > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:705)
11:30:40    > 	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.assertWatchWithMinimumActionsCount(AbstractWatcherIntegrationTestCase.java:456)
11:30:40    > 	at org.elasticsearch.xpack.watcher.history.HistoryActionConditionTests.testActionConditionWithFailures(HistoryActionConditionTests.java:162)
11:30:40    > 	at java.lang.Thread.run(Thread.java:748)

However, higher up the log is this exception:

11:30:40   1> java.lang.IllegalStateException: not started
11:30:40   1> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService.processEventsSync(ExecutionService.java:241) ~[main/:?]
11:30:40   1> 	at org.elasticsearch.xpack.watcher.execution.SyncTriggerEventConsumer.accept(SyncTriggerEventConsumer.java:44) ~[main/:?]
11:30:40   1> 	at org.elasticsearch.xpack.watcher.execution.SyncTriggerEventConsumer.accept(SyncTriggerEventConsumer.java:31) ~[main/:?]
11:30:40   1> 	at org.elasticsearch.xpack.watcher.trigger.TriggerService$GroupedConsumer.lambda$accept$0(TriggerService.java:191) ~[main/:?]
11:30:40   1> 	at java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:890) ~[?:1.8.0_144]
11:30:40   1> 	at org.elasticsearch.xpack.watcher.trigger.TriggerService$GroupedConsumer.accept(TriggerService.java:191) ~[main/:?]
11:30:40   1> 	at org.elasticsearch.xpack.watcher.trigger.TriggerService$GroupedConsumer.accept(TriggerService.java:181) ~[main/:?]
11:30:40   1> 	at org.elasticsearch.xpack.watcher.trigger.ScheduleTriggerEngineMock.lambda$trigger$0(ScheduleTriggerEngineMock.java:113) ~[test/:?]
11:30:40   1> 	at java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:890) ~[?:1.8.0_144]
11:30:40   1> 	at org.elasticsearch.xpack.watcher.trigger.ScheduleTriggerEngineMock.trigger(ScheduleTriggerEngineMock.java:113) ~[test/:?]
11:30:40   1> 	at org.elasticsearch.xpack.watcher.trigger.ScheduleTriggerEngineMock.trigger(ScheduleTriggerEngineMock.java:96) ~[test/:?]
11:30:40   1> 	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase$TimeWarp.lambda$trigger$0(AbstractWatcherIntegrationTestCase.java:560) ~[test/:?]
11:30:40   1> 	at java.util.ArrayList.forEach(ArrayList.java:1249) [?:1.8.0_144]
11:30:40   1> 	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase$TimeWarp.trigger(AbstractWatcherIntegrationTestCase.java:560) [test/:?]
11:30:40   1> 	at org.elasticsearch.xpack.watcher.history.HistoryActionConditionTests.putAndTriggerWatch(HistoryActionConditionTests.java:271) [test/:?]
11:30:40   1> 	at org.elasticsearch.xpack.watcher.history.HistoryActionConditionTests.testActionConditionWithFailures(HistoryActionConditionTests.java:161) [test/:?]
11:30:40   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
11:30:40   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
11:30:40   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
11:30:40   1> 	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_144]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) [lucene-test-framework-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:44]
11:30:40   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:44]
11:30:40   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [lucene-test-framework-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:44]
11:30:40   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:44]
11:30:40   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:44]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [lucene-test-framework-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:44]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) [lucene-test-framework-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:44]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:44]
11:30:40   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [lucene-test-framework-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:44]
11:30:40   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [lucene-test-framework-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:44]
11:30:40   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [lucene-test-framework-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:44]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [randomizedtesting-runner-2.5.2.jar:?]
11:30:40   1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

So it is quite likely that the underlying problem is the same as the first test failure.

[elasticmachine]

Original comment by @imotov:

Another instance LINK REDACTED

[elasticmachine]

Original comment by @martijnvg:

Another instance of:

FAILURE 10.2s J0 | WatcherStatsTests.testStartedStats <<< FAILURES!
   > Throwable LINK REDACTED: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_t0, v2=STOPPED]]
   > Expected: every item is is <STARTED>
   >      but: an item was <STOPPED>
   > 	at __randomizedtesting.SeedInfo.seed([22381737B31B580D:C0F0398A7262856A]:0)
   > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:505)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:731)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:705)
   > 	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.startWatcher(AbstractWatcherIntegrationTestCase.java:491)
   > 	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase._setup(AbstractWatcherIntegrationTestCase.java:210)
   > 	at java.lang.Thread.run(Thread.java:748)
   > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_t0, v2=STOPPED]]
   > Expected: every item is is <STARTED>
   >      but: an item was <STOPPED>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:505)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
   > 		... 38 more
   > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_t0, v2=STOPPED]]
   > Expected: every item is is <STARTED>
   >      but: an item was <STOPPED>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:505)
  2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+x-pack-elasticsearch+6.2+multijob-unix-compatibility/elasticsearch-extra/x-pack-elasticsearch/plugin/watcher/build/testrun/test/J0/temp/org.elasticsearch.xpack.watcher.transport.action.stats.WatcherStatsTests_22381737B31B580D-001
  2> NOTE: test params are: codec=Asserting(Lucene70): {_id=PostingsFormat(name=Asserting)}, docValues:{_seq_no=DocValuesFormat(name=Asserting), _primary_term=DocValuesFormat(name=Asserting), _version=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=1728, maxMBSortInHeap=6.962151792971328, sim=RandomSimilarity(queryNorm=false): {}, locale=en-ZA, timezone=SystemV/EST5EDT
  2> NOTE: Linux 4.4.0-1048-aws amd64/Oracle Corporation 1.8.0_162 (64-bit)/cpus=4,threads=1,free=342458688,total=511180800
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
   > 		... 38 more
   > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_t0, v2=STOPPED]]
  2> NOTE: All tests run in this JVM: [DeleteWatchTests, HistoryTemplateTransformMappingsTests, HttpRequestTemplateTests, OldWatcherStatsTests, HttpInputTests, TimeThrottleIntegrationTests, HistoryStoreTests, ManualExecutionTests, GetWatchTests, ScriptTransformTests, ExecutableSlackActionTests, MonthlyScheduleTests, DataAttachmentParserTests, LoggingActionTests, WatchMetadataTests, NotificationServiceTests, CronEvalToolTests, WatchTests, AckThrottlerTests, HttpSecretsIntegrationTests, EmailSecretsIntegrationTests, XContentSourceTests, WeeklyScheduleTests, SlackMessageTests, CompareConditionSearchTests, ChainIntegrationTests, WatcherServiceTests, WatcherUtilsTests, AccountTests, DailyScheduleTests, HistoryTemplateIndexActionMappingsTests, HttpClientTests, ExecutableJiraActionTests, HipChatServiceTests, AlwaysConditionTests, WatcherMetaDataSerializationTests, ExecutionVarsIntegrationTests, WatcherFeatureSetTests, WatcherIndexingListenerTests, HipChatActionFactoryTests, HipChatActionTests, IndexActionTests, JiraActionTests, WebhookActionTests, WebhookHttpsIntegrationTests, SizeLimitInputStreamTests, CountersTests, NeverConditionTests, ScriptConditionSearchTests, ScriptConditionTests, ExecutionServiceTests, HistoryTemplateHttpMappingsTests, HistoryTemplateTimeMappingsTests, ChainInputTests, EmailTests, HtmlSanitizerTests, ReportingAttachmentParserTests, HipChatAccountsTests, V1AccountTests, JiraAccountTests, IncidentEventDefaultsTests, PagerDutyAccountsTests, VariablesTests, WatcherDateTimeUtilsTests, MapPathTests, HistoryIntegrationTests, SearchTransformTests, SlackServiceTests, ChainTransformTests, WatcherStatsTests]
   > Expected: every item is is <STARTED>
   >      but: an item was <STOPPED>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:505)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
   > 		... 38 more
   > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_t0, v2=STOPPED]]
   > Expected: every item is is <STARTED>
   >      but: an item was <STOPPED>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:505)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
   > 		... 38 more
   > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_t0, v2=STOPPED]]
   > Expected: every item is is <STARTED>
   >      but: an item was <STOPPED>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:505)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
   > 		... 38 more
   > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_t0, v2=STOPPED]]
   > Expected: every item is is <STARTED>
   >      but: an item was <STOPPED>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:505)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
   > 		... 38 more
   > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_t0, v2=STOPPED]]
   > Expected: every item is is <STARTED>
   >      but: an item was <STOPPED>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:505)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
   > 		... 38 more
   > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_t0, v2=STOPPED]]
   > Expected: every item is is <STARTED>
   >      but: an item was <STOPPED>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:505)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
   > 		... 38 more
   > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_t0, v2=STOPPED]]
   > Expected: every item is is <STARTED>
   >      but: an item was <STOPPED>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:505)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
   > 		... 38 more
   > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_t0, v2=STOPPED]]
   > Expected: every item is is <STARTED>
   >      but: an item was <STOPPED>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:505)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
   > 		... 38 more
   > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_t0, v2=STOPPED]]
   > Expected: every item is is <STARTED>
   >      but: an item was <STOPPED>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:505)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
   > 		... 38 more
   > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_t0, v2=STOPPED]]
   > Expected: every item is is <STARTED>
   >      but: an item was <STOPPED>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:505)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
   > 		... 38 more
   > 	Suppressed: java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_t0, v2=STOPPED]]
   > Expected: every item is is <STARTED>
   >      but: an item was <STOPPED>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:505)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
   > 		... 38 more
Completed [134/146] on J0 in 11.28s, 2 tests, 1 failure <<< FAILURES!

build url: LINK REDACTED

[elasticmachine]

Original comment by @davidkyle:

A similar failure but this time in the test _ testWatchMetadata_

java.lang.AssertionError: Expected watcher to be started, but state was [Tuple [v1=node_s0, v2=STARTED], Tuple [v1=node_s2, v2=STOPPED], Tuple [v1=node_s1, v2=STARTED]]
Expected: every item is is <STARTED>
     but: an item was <STOPPED>
	at __randomizedtesting.SeedInfo.seed([25E4436978867020:B29CD2A2860CE812]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$startWatcher$6(AbstractWatcherIntegrationTestCase.java:473)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:734)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:708)

LINK REDACTED

Does not reproduce

gradlew :x-pack-elasticsearch:plugin:watcher:test \
  -Dtests.seed=25E4436978867020 \
  -Dtests.class=org.elasticsearch.xpack.watcher.test.integration.WatchMetadataTests \
  -Dtests.method="testWatchMetadata" \
  -Dtests.security.manager=true \
  -Dtests.locale=sk \
  -Dtests.timezone=America/Rosario

[elasticmachine]

Original comment by @albertzaharovits:

Another failure as Dave pointed out in LINK REDACTED :
LINK REDACTED

14:25:28   1> [2018-04-11T14:25:28,484][INFO ][o.e.x.w.h.HistoryActionConditionTests] [testActionCondition]: after test
14:25:28 FAILURE 10.7s J2 | HistoryActionConditionTests.testActionCondition <<< FAILURES!
14:25:28    > Throwable LINK REDACTED: java.lang.AssertionError: could not find executed watch record
14:25:28    > Expected: a value equal to or greater than <1L>
14:25:28    >      but: <0L> was less than <1L>
14:25:28    > 	at __randomizedtesting.SeedInfo.seed([AE7E919DDB68D309:5ED6F328BD05BE8F]:0)
14:25:28    > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
14:25:28    > 	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$assertWatchWithMinimumActionsCount$2(AbstractWatcherIntegrationTestCase.java:453)
14:25:28    > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:731)
14:25:28    > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:705)
14:25:28    > 	at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.assertWatchWithMinimumActionsCount(AbstractWatcherIntegrationTestCase.java:437)
14:25:28    > 	at org.elasticsearch.xpack.watcher.history.HistoryActionConditionTests.testActionCondition(HistoryActionConditionTests.java:212)
14:25:28    > 	at java.lang.Thread.run(Thread.java:748)
14:25:28    > 	Suppressed: java.lang.AssertionError: could not find executed watch record
14:25:28    > Expected: a value equal to or greater than <1L>
14:25:28    >      but: <0L> was less than <1L>
14:25:28    > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
14:25:28    > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$assertWatchWithMinimumActionsCount$2(AbstractWatcherIntegrationTestCase.java:453)
14:25:28    > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
14:25:28    > 		... 40 more
14:25:28    > 	Suppressed: java.lang.AssertionError: could not find executed watch record
14:25:28    > Expected: a value equal to or greater than <1L>
14:25:28    >      but: <0L> was less than <1L>
14:25:28    > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
14:25:28    > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$assertWatchWithMinimumActionsCount$2(AbstractWatcherIntegrationTestCase.java:453)
14:25:28    > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
14:25:28    > 		... 40 more
14:25:28    > 	Suppressed: java.lang.AssertionError: could not find executed watch record
14:25:28    > Expected: a value equal to or greater than <1L>
14:25:28    >      but: <0L> was less than <1L>
14:25:28    > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
14:25:28    > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$assertWatchWithMinimumActionsCount$2(AbstractWatcherIntegrationTestCase.java:453)
14:25:28    > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
14:25:28    > 		... 40 more
14:25:28    > 	Suppressed: java.lang.AssertionError: could not find executed watch record
14:25:28    > Expected: a value equal to or greater than <1L>
14:25:28    >      but: <0L> was less than <1L>
14:25:28    > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
14:25:28    > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$assertWatchWithMinimumActionsCount$2(AbstractWatcherIntegrationTestCase.java:453)
14:25:28    > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
14:25:28    > 		... 40 more
14:25:28    > 	Suppressed: java.lang.AssertionError: could not find executed watch record
14:25:28    > Expected: a value equal to or greater than <1L>
14:25:28    >      but: <0L> was less than <1L>
14:25:29    > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
14:25:29    > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$assertWatchWithMinimumActionsCount$2(AbstractWatcherIntegrationTestCase.java:453)
14:25:29    > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
14:25:29    > 		... 40 more
14:25:29    > 	Suppressed: java.lang.AssertionError: could not find executed watch record
14:25:29    > Expected: a value equal to or greater than <1L>
14:25:29    >      but: <0L> was less than <1L>
14:25:29    > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
14:25:29    > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$assertWatchWithMinimumActionsCount$2(AbstractWatcherIntegrationTestCase.java:453)
14:25:29    > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
14:25:29    > 		... 40 more
14:25:29    > 	Suppressed: java.lang.AssertionError: could not find executed watch record
14:25:29    > Expected: a value equal to or greater than <1L>
14:25:29    >      but: <0L> was less than <1L>
14:25:29    > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
14:25:29    > 		at org.elasticsearch.xpack.watcher.test.AbstractWatcherIntegrationTestCase.lambda$assertWatchWithMinimumActionsCount$2(AbstractWatcherIntegrationTestCase.java:453)
14:25:29    > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:719)
14:25:29    > 		... 40 more
14:25:29   2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+x-pack-elasticsearch+6.2+release-tests/elasticsearch-extra/x-pack-elasticsearch/plugin/watcher/build/testrun/test/J2/temp/org.elasticsearch.xpack.watcher.history.HistoryActionConditionTests_AE7E919DDB68D309-001
14:25:29   2> NOTE: test params are: codec=Asserting(Lucene70): {result.actions.condition.type.keyword=PostingsFormat(name=Asserting), 

[elasticmachine]

Original comment by @javanna:

Another occurrence: LINK REDACTED

[jasontedor]

Closed by #30118