[CI] org.elasticsearch.xpack.ilm.TimeseriesMoveToStepIT.testMoveToRolloverStep fails #85036
Description
CI Link
https://gradle-enterprise.elastic.co/s/deqtxspnhxyva
Repro line
./gradlew ':x-pack:plugin:ilm:qa:multi-node:javaRestTest' --tests "org.elasticsearch.xpack.ilm.TimeseriesMoveToStepIT.testMoveToRolloverStep" -Dtests.seed=81FF6263145E4101 -Dtests.locale=es-ES -Dtests.timezone=America/Godthab -Druntime.java=8
Does it reproduce?
No
Applicable branches
7.17
Failure history
Failed two times in the last 30 days, all very recent: https://gradle-enterprise.elastic.co/s/yp56iltyyinha
Failure excerpt
The tests TimeseriesMoveToStepIT.testMoveToRolloverStep and TimeseriesMoveToStepIT.testMoveToAllocateStep both failed.
I'm not sure what is the root cause but this is maybe interesting:
» [2022-03-16T10:55:00,346][ERROR][o.e.ExceptionsHelper ] [javaRestTest-0] fatal error
» at org.elasticsearch.ExceptionsHelper.lambda$maybeDieOnAnotherThread$4(ExceptionsHelper.java:287)
» at java.util.Optional.ifPresent(Optional.java:159)
» at org.elasticsearch.ExceptionsHelper.maybeDieOnAnotherThread(ExceptionsHelper.java:277)
» at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:229)
» at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
» at java.util.concurrent.FutureTask.run(FutureTask.java:266)
» at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
» at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
» at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
» at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
» at java.lang.Thread.run(Thread.java:748)
» [2022-03-16T10:55:00,347][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [javaRestTest-0] fatal error in thread [Thread-4], exiting
» java.lang.AssertionError: policy step registry cache failed sanity check
» at org.elasticsearch.xpack.ilm.PolicyStepsRegistry.getStep(PolicyStepsRegistry.java:378) ~[?:?]
» at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.getCurrentStep(IndexLifecycleRunner.java:108) ~[?:?]
» at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:173) ~[?:?]
» at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:418) ~[?:?]
» at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:349) ~[?:?]
» at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:186) ~[?:?]
» at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:220) ~[?:?]
» at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_301]
» at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_301]
» at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_301]
» at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_301]
» at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_301]
» at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_301]
» at java.lang.Thread.run(Thread.java:748) [?:1.8.0_301]
» ↓ last 40 non err
As well as :
"action":"init","step":"ERROR","step_time":"1647428098374"},"error_details":"{\"type\":\"initialize_policy_exception\",\"reason\":\"unable to initialize policy [policy-zZOnT] for index [retryinit-20xx-01-10]\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"index name [retryinit-20xx-01-10] does not match pattern
The test ShrinkActionIT.testShrinkDuringSnapshot also failed but I suspect it is more a consequence of the other failures.