testRandomDirectoryIOExceptions fails (on PRRLs branch) #45072
Description
SearchWithRandomIOExceptionsIT.testRandomDirectoryIOExceptions fails on the peer-recovery-retention-leases branch (6960cf7) with a number of assertion errors of the following form:
2> java.lang.AssertionError: starting seq no is set to 0 but history uuid is unset
2> at __randomizedtesting.SeedInfo.seed([8931119804B52E1]:0)
2> at org.elasticsearch.indices.recovery.StartRecoveryRequest.<init>(StartRecoveryRequest.java:86)
2> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.getStartRecoveryRequest(PeerRecoveryTargetService.java:356)
2> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:179)
2> at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:553)
2> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:769)
2> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2> at java.base/java.lang.Thread.run(Thread.java:835)
Reproduce with:
./gradlew :server:integTest --tests "org.elasticsearch.search.basic.SearchWithRandomIOExceptionsIT.testRandomDirectoryIOExceptions" -Dtests.seed=8931119804B52E1 -Dtests.security.manager=true -Dtests.jvms=4 -Dtests.locale=ha-NG -Dtests.timezone=Asia/Aden -Dcompiler.java=12 -Druntime.java=12
It looks like a recovery is completing phase 1 but then failing before establishing a history UUID, but a subsequent recovery is using the existing data to recover locally anyway.