[CI] ExplainLifecycleIT testExplainFilters failure

[gwbrown]

Build scan: https://gradle-enterprise.elastic.co/s/hc2lfdckt3nf4

Branch: PR (#70887) from 7.x with unrelated changes

Repro line (reproduces locally, unreliably, more on slower machines seems like):

./gradlew ':x-pack:plugin:ilm:qa:multi-node:javaRestTest' --tests "org.elasticsearch.xpack.ilm.ExplainLifecycleIT.testExplainFilters" -Dtests.seed=BCA33ACEFE7E5C8E -Dtests.security.manager=true -Dtests.locale=es-CO -Dtests.timezone=Africa/Windhoek -Druntime.java=8

Stack trace:

java.lang.AssertionError: 	
Expected: (map containing ["index-nejuhwmcvh-error"->ANYTHING] and map containing ["index-nejuhwmcvh-nonexistant-policy"->ANYTHING])	
     but: map containing ["index-nejuhwmcvh-error"->ANYTHING] map was [<index-nejuhwmcvh-nonexistant-policy={index=index-nejuhwmcvh-nonexistant-policy, managed=true, policy=McUjQJBj, step_info={type=illegal_argument_exception, reason=policy [McUjQJBj] does not exist}}>]	
	at __randomizedtesting.SeedInfo.seed([BCA33ACEFE7E5C8E:496CC337BF62963F]:0)	
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)	
	at org.junit.Assert.assertThat(Assert.java:956)	
	at org.junit.Assert.assertThat(Assert.java:923)	
	at org.elasticsearch.xpack.ilm.ExplainLifecycleIT.lambda$testExplainFilters$1(ExplainLifecycleIT.java:117)	
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1017)	
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:990)	
	at org.elasticsearch.xpack.ilm.ExplainLifecycleIT.testExplainFilters(ExplainLifecycleIT.java:104)	
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)	
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)	
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)	
	at java.lang.reflect.Method.invoke(Method.java:498)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)	
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)	
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)	
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)	
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)	
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)	
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)	
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)	
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)	
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)	
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)	
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)	
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)	
	at java.lang.Thread.run(Thread.java:748)

Failure history: Only failed in that build and locally on my macbook.

@andreidan I assigned you because Lee let me know that this might be related to some recent work of yours.

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[ywelsch]

Another failure: https://gradle-enterprise.elastic.co/s/fwqiomma5rm5w/tests/:x-pack:plugin:ilm:qa:multi-node:javaRestTest/org.elasticsearch.xpack.ilm.ExplainLifecycleIT/testExplainFilters#1

[romseygeek]

Another failure on a PR here: https://gradle-enterprise.elastic.co/s/ffaskcunkpy3e/tests/:x-pack:plugin:ilm:qa:multi-node:javaRestTest/org.elasticsearch.xpack.ilm.ExplainLifecycleIT/testExplainFilters

[mayya-sharipova]

Another failure on a PR here: https://gradle-enterprise.elastic.co/s/dwjrg6w2oherw

java.lang.AssertionError: 	
Expected: (map containing ["index-tclbenjtxt-error"->ANYTHING] and map containing ["index-tclbenjtxt-nonexistant-policy"->ANYTHING])	
     but: map containing ["index-tclbenjtxt-error"->ANYTHING] map was [<index-tclbenjtxt-nonexistant-policy={index=index-tclbenjtxt-nonexistant-policy, managed=true, policy=bioJfmeEm, step_info={type=illegal_argument_exception, reason=policy [bioJfmeEm] does not exist}}>]	
at __randomizedtesting.SeedInfo.seed([9127FBF119B6BF7F:64E8020858AA75CE]:0)	
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)	
at org.junit.Assert.assertThat(Assert.java:956)	
at org.junit.Assert.assertThat(Assert.java:923)	
at org.elasticsearch.xpack.ilm.ExplainLifecycleIT.lambda$testExplainFilters$1(ExplainLifecycleIT.java:117)

[benwtrent]

Exact same failure again.

https://gradle-enterprise.elastic.co/s/yli373x6auyem/

org.elasticsearch.xpack.ilm.ExplainLifecycleIT > testExplainFilters FAILED |
-- | --
  | java.lang.AssertionError: |
  | Expected: (map containing ["index-tfdcxzfsrr-error"->ANYTHING] and map containing ["index-tfdcxzfsrr-nonexistant-policy"->ANYTHING]) |
  | but: map containing ["index-tfdcxzfsrr-error"->ANYTHING] map was [<index-tfdcxzfsrr-nonexistant-policy={index=index-tfdcxzfsrr-nonexistant-policy, managed=true, policy=FklioxsVv, step_info={type=illegal_argument_exception, reason=policy [FklioxsVv] does not exist}}>] |
  | at __randomizedtesting.SeedInfo.seed([AA6923D353468540:5FA6DA2A125A4FF1]:0) |
  | at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) |
  | at org.junit.Assert.assertThat(Assert.java:956) |
  | at org.junit.Assert.assertThat(Assert.java:923) |
  | at org.elasticsearch.xpack.ilm.ExplainLifecycleIT.lambda$testExplainFilters$1(ExplainLifecycleIT.java:117) |
  | at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:973) |
  | at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:946) |
  | at org.elasticsearch.xpack.ilm.ExplainLifecycleIT.testExplainFilters(ExplainLifecycleIT.java:104)

It happens fairly rarely, 2-3 times a week