SnapshotLifecycleTemplateRegistryTests.testThatNonExistingTemplatesAreAddedImmediately fails on CI non-reproducibly

[gwbrown]

Build scan from a PR that changes this test, but it fails in a similar way unchanged as well.

Reproduce line (does not reproduce locally):

./gradlew :x-pack:plugin:core:test --tests "org.elasticsearch.xpack.core.snapshotlifecycle.history.SnapshotLifecycleTemplateRegistryTests.testThatNonExistingTemplatesAreAddedImmediately" -Dtests.seed=1F3A88753168CB4 -Dtests.security.manager=true -Dtests.locale=hsb -Dtests.timezone=America/Boise -Dcompiler.java=12 -Druntime.java=11

Stack trace:

java.lang.AssertionError: 
expected latch to be counted down after 10s, but was not
Expected: is <true>
     but: was <false>
at __randomizedtesting.SeedInfo.seed([1F3A88753168CB4:6AF45AD77CF822B5]:0)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.awaitLatch(ElasticsearchAssertions.java:686)
at org.elasticsearch.xpack.core.snapshotlifecycle.history.SnapshotLifecycleTemplateRegistryTests.testThatNonExistingTemplatesAreAddedImmediately(SnapshotLifecycleTemplateRegistryTests.java:145)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:566)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:834)

I've tried increasing the timeout to 60s but that doesn't prevent failures. This one is going to need some more thorough investigation. There's a very similar test for Watcher's index templates, which uses a mocked threadpool that runs client invocations on the same thread as the test, which is very stable. This test uses a real threadpool and waits for the call to happen in the background, but is otherwise extremely similar, so I suspect there's just an issue with the test.

[elasticmachine]

Pinging @elastic/es-core-features

[gwbrown]

I've deleted a spam comment that was posted to this issue with a very sketchy-looking link.

[cbuescher]

This failed again today on master after the unmute, so reopening:
https://gradle-enterprise.elastic.co/s/gzcjmzrzdbbl2/console-log?task=:x-pack:plugin:core:test

[andreidan]

Ah it's a different failure this time, looking into it