[CI] Failure in org.elasticsearch.snapshots.SnapshotResiliencyTests.testSnapshotWithNodeDisconnects

[original-brownbear]

Failed here: https://gradle-enterprise.elastic.co/s/tr5z6fea45tsu/console-log#L2589

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.elasticsearch.snapshots.SnapshotResiliencyTests.testSnapshotWithNodeDisconnects" -Dtests.seed=C9164645C24DC7E4 -Dtests.security.manager=true -Dtests.locale=ti -Dtests.timezone=Asia/Novosibirsk -Dcompiler.java=12 -Druntime.java=11

fails with

org.elasticsearch.snapshots.SnapshotResiliencyTests > testSnapshotWithNodeDisconnects FAILED
--
java.lang.AssertionError:
Expected: map containing ["snap-atluIEEOR0-Jg-L2LxO3wg.dat"->ANYTHING]
but: map was []
at __randomizedtesting.SeedInfo.seed([C9164645C24DC7E4:C7D9E98DD88BFC24]:0)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at org.elasticsearch.repositories.blobstore.BlobStoreTestUtil.assertSnapshotUUIDs(BlobStoreTestUtil.java:175)
at org.elasticsearch.repositories.blobstore.BlobStoreTestUtil$1.doRun(BlobStoreTestUtil.java:110)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.repositories.blobstore.BlobStoreTestUtil.assertConsistency(BlobStoreTestUtil.java:92)
at org.elasticsearch.snapshots.SnapshotResiliencyTests.verifyReposThenStopServices(SnapshotResiliencyTests.java:238)

I can reproduce this locally when running the test in a loop with the given seed.
This means two problems:

1. The test fails
2. The test is supposed to be fully deterministic but apparently it's not since it doesn't reliably fail with the given seed.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

[ywelsch]

Try using -Dhppc.bitmixer=DETERMINISTIC to see if that makes the failures deterministic (it's the same trick we have to use for the CoordinatorTests

[original-brownbear]

Thanks @ywelsch that does indeed make things deterministic (albeit in this case deterministically passing :)).