[CI] SLMSnapshotBlockingIntegTests testRetentionWithMultipleRepositories failing

[mark-vieira]

This is a newly added test that has already failed a number of times in CI.

Build scan:
https://gradle-enterprise.elastic.co/s/ffmncmzt23eay/tests/:x-pack:plugin:ilm:internalClusterTest/org.elasticsearch.xpack.slm.SLMSnapshotBlockingIntegTests/testRetentionWithMultipleRepositories#1

Repro line:
./gradlew ':x-pack:plugin:ilm:internalClusterTest' --tests "org.elasticsearch.xpack.slm.SLMSnapshotBlockingIntegTests.testRetentionWithMultipleRepositories" -Dtests.seed=9BFBC3AF7F9C991C -Dtests.security.manager=true -Dtests.locale=es-DO -Dtests.timezone=SST -Druntime.java=11

Reproduces locally?:
Nope.

Applicable branches:
7.11, 7.12, 7.x and master

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?search.relativeStartTime=P7D&search.timeZoneId=America/Los_Angeles&tests.container=org.elasticsearch.xpack.slm.SLMSnapshotBlockingIntegTests&tests.disabledDistributions=WyJvdXRjb21lOnBhc3NlZCIsIm91dGNvbWU6c2tpcHBlZCJd&tests.sortField=FAILED&tests.test=testRetentionWithMultipleRepositories&tests.unstableOnly=true

Failure excerpt:

org.elasticsearch.xpack.slm.SLMSnapshotBlockingIntegTests > testRetentionWithMultipleRepositories FAILED
    java.lang.AssertionError: timed out waiting for green state
        at __randomizedtesting.SeedInfo.seed([9BFBC3AF7F9C991C:35EAB13BB52B8935]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.elasticsearch.test.ESIntegTestCase.ensureColor(ESIntegTestCase.java:903)
        at org.elasticsearch.test.ESIntegTestCase.ensureGreen(ESIntegTestCase.java:856)
        at org.elasticsearch.test.ESIntegTestCase.ensureGreen(ESIntegTestCase.java:845)
        at org.elasticsearch.xpack.slm.SLMSnapshotBlockingIntegTests.createAndPopulateIndex(SLMSnapshotBlockingIntegTests.java:465)
        at org.elasticsearch.xpack.slm.SLMSnapshotBlockingIntegTests.testUnsuccessfulSnapshotRetention(SLMSnapshotBlockingIntegTests.java:349)
        at org.elasticsearch.xpack.slm.SLMSnapshotBlockingIntegTests.testRetentionWithMultipleRepositories(SLMSnapshotBlockingIntegTests.java:280)

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[mark-vieira]

@original-brownbear do you mind taking a look since you recently added this test?

FYI, I've muted this in all branches (master, 7.x, 7.12, and 7.11).

[original-brownbear]

Sure thing @mark-vieira I opened #70202 that should deal with this one