[CI] XDocsClientYamlTestSuiteIT failed because of no `.watches` index

[DaveCTurner]

Two seemingly-unrelated tests failed for the same reason in the same run:

   > Throwable #1: java.lang.AssertionError: 4 active tasks found:
   > indices:data/write/update      YJDcklJoRF-6apksAvIohw:845 -                          transport 1539506880310 08:48:00 10.4s       127.0.0.1 node-0 
   > indices:data/write/update[s]   YJDcklJoRF-6apksAvIohw:846 -                          direct    1539506880321 08:48:00 10.4s       127.0.0.1 node-0 
   > indices:data/write/bulk        YJDcklJoRF-6apksAvIohw:847 -                          transport 1539506880322 08:48:00 10.4s       127.0.0.1 node-0 requests[1], indices[.watches]
   > indices:data/write/bulk[s]     YJDcklJoRF-6apksAvIohw:848 YJDcklJoRF-6apksAvIohw:847 transport 1539506880322 08:48:00 10.4s       127.0.0.1 node-0 requests[1], index[.watches]
   >  expected:<0> but was:<4>
   > 	at __randomizedtesting.SeedInfo.seed([8E320CEFCF5C0656:666333561A06BAE]:0)
   > 	at org.elasticsearch.xpack.test.rest.XPackRestTestHelper.lambda$waitForPendingTasks$3(XPackRestTestHelper.java:134)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:847)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:821)
   > 	at org.elasticsearch.xpack.test.rest.XPackRestTestHelper.waitForPendingTasks(XPackRestTestHelper.java:110)
   > 	at org.elasticsearch.xpack.test.rest.XPackRestTestHelper.waitForPendingTasks(XPackRestTestHelper.java:97)
   > 	at org.elasticsearch.xpack.test.rest.XPackRestIT.cleanup(XPackRestIT.java:246)
   > 	at java.lang.Thread.run(Thread.java:748)
   > 	Suppressed: java.lang.AssertionError: 4 active tasks found:
   > indices:data/write/update      YJDcklJoRF-6apksAvIohw:845 -                          transport 1539506880310 08:48:00 410ms       127.0.0.1 node-0 
   > indices:data/write/update[s]   YJDcklJoRF-6apksAvIohw:846 -                          direct    1539506880321 08:48:00 399.1ms     127.0.0.1 node-0 
   > indices:data/write/bulk        YJDcklJoRF-6apksAvIohw:847 -                          transport 1539506880322 08:48:00 397.9ms     127.0.0.1 node-0 requests[1], indices[.watches]
   > indices:data/write/bulk[s]     YJDcklJoRF-6apksAvIohw:848 YJDcklJoRF-6apksAvIohw:847 transport 1539506880322 08:48:00 397.8ms     127.0.0.1 node-0 requests[1], index[.watches]
   >  expected:<0> but was:<4>
   > 		at org.elasticsearch.xpack.test.rest.XPackRestTestHelper.lambda$waitForPendingTasks$3(XPackRestTestHelper.java:134)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:835)
   > 		... 38 more

The server logs indicate that the .watches index does not exist:

[2018-10-14T08:48:21,707][WARN ][o.e.a.b.TransportShardBulkAction] [node-0] unexpected error during the primary phase for action [indices:data/write/bulk[s]], request [BulkShardRequest [[.watches][0]] containing [index {[.watches][doc][my-watch], source[{"trigger":{"schedule":{"cron":"0 0/1 * * * ?"}},"input":{"search":{"request":{"search_type":"query_then_fetch","indices":["logstash*"],"types":[],"body":{"query":{"bool":{"filter":{"range":{"@timestamp":{"from":"{{ctx.trigger.scheduled_time}}||-5m","to":"{{ctx.trigger.triggered_time}}"}}},"must":{"match":{"response":404}}}}}}}},"condition":{"compare":{"ctx.payload.hits.total":{"gt":0}}},"actions":{"email_admin":{"email":{"profile":"standard","to":["admin@domain.host.com"],"subject":"404 recently encountered"}}},"status":{"state":{"active":true,"timestamp":"2018-10-14T08:47:59.870Z"},"actions":{"email_admin":{"ack":{"timestamp":"2018-10-14T08:47:59.870Z","state":"awaits_successful_execution"}}},"headers":{"_xpack_security_authentication":"s6LxAgAKdGVzdF9hZG1pbgEJc3VwZXJ1c2VyCgAAAAEABm5vZGUtMAxkZWZhdWx0X2ZpbGUEZmlsZQA="},"version":-1,"last_checked":"2018-10-14T08:48:00.169Z","execution_state":"execution_not_needed"}}]}]]
org.elasticsearch.index.IndexNotFoundException: no such index
	at org.elasticsearch.cluster.routing.RoutingTable.shardRoutingTable(RoutingTable.java:137) ~[elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.primary(TransportReplicationAction.java:795) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:730) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onNewClusterState(TransportReplicationAction.java:881) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onNewClusterState(ClusterStateObserver.java:303) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.clusterChanged(ClusterStateObserver.java:191) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateListeners$7(ClusterApplierService.java:492) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at java.util.concurrent.ConcurrentHashMap$KeySpliterator.forEachRemaining(ConcurrentHashMap.java:3566) [?:?]
	at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735) [?:?]
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) [?:?]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:489) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:472) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:416) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:160) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.5.0-SNAPSHOT.jar:6.5.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]

The run was
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=amazon/60/console and apparently this can be reproduced with

REPRODUCE WITH: ./gradlew :x-pack:docs:integTestRunner \
  -Dtests.seed=8E320CEFCF5C0656 \
  -Dtests.class=org.elasticsearch.smoketest.XDocsClientYamlTestSuiteIT \
  -Dtests.method="test {yaml=en/security/authorization/role-templates/line_53}" \
  -Dtests.security.manager=true \
  -Dtests.locale=es-UY \
  -Dtests.timezone=America/Argentina/Tucuman \
  -Dcompiler.java=11 \
  -Druntime.java=8

REPRODUCE WITH: ./gradlew :x-pack:docs:integTestRunner \
  -Dtests.seed=8E320CEFCF5C0656 \
  -Dtests.class=org.elasticsearch.smoketest.XDocsClientYamlTestSuiteIT \
  -Dtests.method="test {yaml=en/security/authentication/saml-guide/line_636}" \
  -Dtests.security.manager=true \
  -Dtests.locale=es-UY \
  -Dtests.timezone=America/Argentina/Tucuman \
  -Dcompiler.java=11 \
  -Druntime.java=8

However this does not reproduce for me.

I searched for similar issues and found #34013 and #33864 which relate to the security index. Perhaps this is similar? Pinging @nik9000, since you're looking at X-pack test cleanup code?

[elasticmachine]

Pinging @elastic/es-core-infra

[nik9000]

I'll add it to the pile, yeah.