WatcherYamlRestIT fails on test get watch api with missing watch on existing index

[DaveCTurner]

The following two builds failed for similar-looking reasons:

• https://gradle-enterprise.elastic.co/s/i6nku22fs7zws
• https://gradle-enterprise.elastic.co/s/jrj3kx4dqgu4w

The failure looked like this:

org.elasticsearch.smoketest.WatcherYamlRestIT > test {p0=watcher/put_watch/10_basic/Test put watch api} FAILED |  
-- | --
  | java.lang.AssertionError: |  
  | Expected: <0> |  
  | but: was <1> |  
  | at __randomizedtesting.SeedInfo.seed([BC95DB412F8CDAF0:34C1E49B8170B708]:0) |  
  | at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) |  
  | at org.junit.Assert.assertThat(Assert.java:956) |  
  | at org.junit.Assert.assertThat(Assert.java:923) |  
  | at org.elasticsearch.xpack.watcher.WatcherYamlSuiteTestCase.lambda$stopWatcher$1(WatcherYamlSuiteTestCase.java:74) |  
  | at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:906) |  
  | at org.elasticsearch.xpack.watcher.WatcherYamlSuiteTestCase.stopWatcher(WatcherYamlSuiteTestCase.java:67) |  
  | at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) |  
  | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) |  
  | at java.lang.reflect.Method.invoke(Method.java:498) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) |  
  | at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) |  
  | at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) |  
  | at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) |  
  | at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) |  
  | at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) |  
  | at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) |  
  | at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) |  
  | at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) |  
  | at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) |  
  | at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) |  
  | at java.lang.Thread.run(Thread.java:748) |  
  | REPRODUCE WITH: ./gradlew ':x-pack:plugin:watcher:qa:rest:integTestRunner' --tests "org.elasticsearch.smoketest.WatcherYamlRestIT.test {p0=watcher/put_watch/10_basic/Test put watch api}" -Dtests.seed=BC95DB412F8CDAF0 -Dtests.security.manager=true -Dtests.locale=es-SV -Dtests.timezone=GMT0 -Dcompiler.java=13 |  
  |  

I haven't attempted to reproduce this.

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Watcher)

[mark-vieira]

I tried, but it didn't reproduce locally for me.

[jaymode]

This also failed on 7.x today https://gradle-enterprise.elastic.co/s/izleuckcsczy6/tests/j3inyc2qyhaby-3e5dsrwmfrm6w.

@martijnvg I see you have a commit that mentions this issue. I wonder if there might be a race condition in stopping the TriggerService where the TriggerService is stopped but a watch is added concurrently/after the stop of the triggerservice somehow, maybe by the WatcherIndexingListener?

[martijnvg]

Thanks @jaymode for triaging this failure. I also did some investigating and if watcher is stopped then the watch_count returned by watcher stats api is 0, because all watcher (trigger service) are paused. However if watcher is stopped, a watch can still be added via the api and this increases the watch count in the watcher stats api by 1. This can also happen if for example a write is retried on a .watches shard copy that has watches allocated. The expectation is that if watcher is stopped then that the watch_count is always zero, but this isn't true. The WatcherIndexingListener doesn't check wether watcher is stopped, just whether it should delegate specific watches to the trigger service. I think that WatcherIndexingListener (and maybe other places too) should also check what the watcher_state is and act on it.

I'm not sure whether this problem occurs here, the watcher rest qa module doesn't execute tests parallel and I've not yet found evidence that a watch was added after watcher was stopped.

The commit that I pushed, delays the checking of the watcher stats' watcher_count that happens between tests. Before this was done after watcher was stopped, but now after watcher is stopped and the .watches index has been removed. I think this make sense based on how the watcher stop api works, but this hasn't fixed the test failure.

[williamrandolph]

There's another instance of this failure on 7.x: https://gradle-enterprise.elastic.co/s/xreqekg5chqoy

One failure on WatcherYamlRestIT:

2> REPRODUCE WITH: ./gradlew ':x-pack:plugin:watcher:qa:rest:integTestRunner' --tests "org.elasticsearch.smoketest.WatcherYamlRestIT.test {p0=watcher/put_watch/10_basic/Test empty body is rejected by put watch}" -Dtests.seed=41277B042F438E42 -Dtests.security.manager=true -Dtests.locale=en-US -Dtests.timezone=Asia/Colombo -Dcompiler.java=13 |  
-- | --
  | 2> java.lang.AssertionError: |  
  | Expected: <0> |  
  | but: was <1> |  
  | at __randomizedtesting.SeedInfo.seed([41277B042F438E42:C97344DE81BFE3BA]:0) |  
  | at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) |  
  | at org.junit.Assert.assertThat(Assert.java:956) |  
  | at org.junit.Assert.assertThat(Assert.java:923) |  
  | at org.elasticsearch.xpack.watcher.WatcherYamlSuiteTestCase.lambda$startWatcher$0(WatcherYamlSuiteTestCase.java:57) |  
  | at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:906) |  
  | at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:879) |  
  | at org.elasticsearch.xpack.watcher.WatcherYamlSuiteTestCase.startWatcher(WatcherYamlSuiteTestCase.java:39)
  […]

…and another on SmokeTestWatcherWithSecurityClientYamlTestSuiteIT:

2> REPRODUCE WITH: ./gradlew ':x-pack:plugin:watcher:qa:with-security:integTestRunner' --tests "org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityClientYamlTestSuiteIT.test {yaml=watcher/activate_watch/10_basic/Non existent watch returns 404}" -Dtests.seed=41277B042F438E42 -Dtests.security.manager=true -Dtests.locale=mt-MT -Dtests.timezone=Asia/Karachi -Dcompiler.java=13 |  
-- | --
  | 2> java.lang.AssertionError: |  
  | Expected: <0> |  
  | but: was <1> |  
  | at __randomizedtesting.SeedInfo.seed([41277B042F438E42:C97344DE81BFE3BA]:0) |  
  | at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) |  
  | at org.junit.Assert.assertThat(Assert.java:956) |  
  | at org.junit.Assert.assertThat(Assert.java:923) |  
  | at org.elasticsearch.xpack.watcher.WatcherYamlSuiteTestCase.lambda$startWatcher$0(WatcherYamlSuiteTestCase.java:57) |  
  | at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:906) |  
  | at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:879) |  
  | at org.elasticsearch.xpack.watcher.WatcherYamlSuiteTestCase.startWatcher(WatcherYamlSuiteTestCase.java:39)

[williamrandolph]

One more on SmokeTestWatcherWithSecurityClientYamlTestSuiteIT, master branch: https://gradle-enterprise.elastic.co/s/43mjgxd6sesck

[jakelandis]

It appears that this may be a setup/tear down issue and build-stats shows more test failures than the ones referenced here. As of #53362 all of the REST tests for Watcher are muted and will look into this issue as soon as possible.