[CI] SnapshotStressTestsIT testRandomActivities failing

[DaveCTurner]

    java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([D22937B3B25069BD]:0)
        at org.elasticsearch.snapshots.InFlightShardSnapshotStates.assertGenerationConsistency(InFlightShardSnapshotStates.java:100)
        at org.elasticsearch.snapshots.InFlightShardSnapshotStates.addStateInformation(InFlightShardSnapshotStates.java:69)
        at org.elasticsearch.snapshots.InFlightShardSnapshotStates.forEntries(InFlightShardSnapshotStates.java:54)
        at org.elasticsearch.cluster.SnapshotsInProgress.assertConsistentEntries(SnapshotsInProgress.java:406)
        at org.elasticsearch.cluster.SnapshotsInProgress.<init>(SnapshotsInProgress.java:119)
        at org.elasticsearch.cluster.SnapshotsInProgress.withUpdatedEntriesForRepo(SnapshotsInProgress.java:135)
        at org.elasticsearch.snapshots.SnapshotsService.stateWithoutSnapshot(SnapshotsService.java:1835)
        at org.elasticsearch.repositories.FinalizeSnapshotContext.updatedClusterState(FinalizeSnapshotContext.java:103)
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository$2.apply(BlobStoreRepository.java:1707)
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository$2.apply(BlobStoreRepository.java:1704)
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository$10.execute(BlobStoreRepository.java:2737)
        at org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.execute(MasterService.java:551)
        at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1050)
        at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1015)
        at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:233)
        at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1656)
        at org.elasticsearch.action.ActionListener.run(ActionListener.java:433)
        at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1653)
        at org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1248)
        at org.elasticsearch.action.ActionListener.run(ActionListener.java:433)
        at org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1227)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1570)

Build scan:
https://gradle-enterprise.elastic.co/s/vgv2xu3qpq5qk/tests/:server:internalClusterTest/org.elasticsearch.snapshots.SnapshotStressTestsIT/testRandomActivities

Reproduction line:

gradlew ':server:internalClusterTest' --tests "org.elasticsearch.snapshots.SnapshotStressTestsIT.testRandomActivities" -Dtests.seed=D22937B3B25069BD -Dtests.locale=no -Dtests.timezone=Indian/Mahe -Druntime.java=22

Applicable branches:
main

Reproduces locally?:
Didn't try

Failure history:
Failure dashboard for org.elasticsearch.snapshots.SnapshotStressTestsIT#testRandomActivities

Failure excerpt:

org.elasticsearch.discovery.MasterNotDiscoveredException: (No message provided)

  at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:324)
  at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:389)
  at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:302)
  at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:650)
  at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:917)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  at java.lang.Thread.run(Thread.java:1570)

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

Reproduced for me (once) after a few thousand runs. Curious, because after we fixed #105245 I ran this suite in a loop for ages without any failures, so maybe it's something we've introduced since then. I'm trying again with more logging.

[DaveCTurner]

Perhaps interestingly it looks like there were indeed no failures of this test suite from early Feb (when #105245 was merged) until mid-March: