[CI] ClusterRerouteIT.testDelayWithALargeAmountOfShards timed out waiting for green state

[tlrx]

The test ClusterRerouteIT » testDelayWithALargeAmountOfShards failed today on CI for the 7.x branch with the error:

timed out waiting for green state
java.lang.AssertionError
: 
timed out waiting for green state
Close stacktrace
at __randomizedtesting.SeedInfo.seed([B25CE8847D0ACBC2:9E2DE2656C14E120]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.elasticsearch.test.ESIntegTestCase.ensureColor(ESIntegTestCase.java:929)
at org.elasticsearch.test.ESIntegTestCase.ensureGreen(ESIntegTestCase.java:882)
at org.elasticsearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards(ClusterRerouteIT.java:224)

Build scan: https://gradle-enterprise.elastic.co/s/3l2rtho4qznu6

Looking at the build stats it also fails on 7.6 (https://gradle-enterprise.elastic.co/s/bxlllkb4jubxs) and seems to be related to the new way of persisting metadata state (see #50907), in which case tests might need to be adapted?

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Cluster Coordination)

[pugnascotia]

Failed again on 7.6 - https://gradle-enterprise.elastic.co/s/wmy4bjcpxqnte

[henningandersen]

Failed again on master: https://gradle-enterprise.elastic.co/s/getynn6hnzfm2

Looked a bit into it, looks like retention lease sync causes the shards to become stale one by one, not sure if this is part of why it is timing out, but it is at least a significant difference since this test was written.

Did not reproduce locally.

[DaveCTurner]

Hmm. All 7 of the failures of this test over the last 30 days were using org.apache.lucene.mockfile.WindowsFS. It typically completes in 30-60 seconds but the failures are 6-8 minutes long. WindowsFS adds extra validation that we behave correctly on filesystems that don't let you delete open files; I wonder if this is introducing extra slowness.

[DaveCTurner]

I opened #52488 to suppress WindowsFS on this test suite.

I am not marking #52488 as closes #52000 since I am not fully certain that this is the underlying problem. I would prefer to have a couple of weeks without failures before declaring this issue closed.