RetentionLeaseSyncIT.testRetentionLeasesSyncOnRecovery fails reproducibly on master and 6.x

[jkakavas]

This looks like it was introduced in #38435. I hit it in a PR build, bit it reproduces consistently on master for me and looking through build failures it has been failing for the last few hours:

1:59:11 FAILURE 0.55s J4 | RetentionLeaseSyncIT.testRetentionLeasesSyncOnRecovery <<< FAILURES!
01:59:11    > Throwable #1: java.lang.AssertionError: shard [index][0], node[Ar2fOmwrR6OAWLivoCoGkA], relocating [J7fySsp9S8KPoeJf_8_q0w], [P], s[RELOCATING], a[id=dmm314dRQ0e7iieuOxH2gw, rId=x0Mzax-eStCxDuQdB39dJg] is not a primary shard in primary mode
01:59:11    > 	at __randomizedtesting.SeedInfo.seed([51682F94E41D8601:1B7C4AFD6CF749F6]:0)
01:59:11    > 	at org.elasticsearch.index.shard.IndexShard.assertPrimaryMode(IndexShard.java:1580)
01:59:11    > 	at org.elasticsearch.index.shard.IndexShard.renewRetentionLease(IndexShard.java:1954)
01:59:11    > 	at org.elasticsearch.index.seqno.RetentionLeaseSyncIT.testRetentionLeasesSyncOnRecovery(RetentionLeaseSyncIT.java:247)
01:59:11    > 	at java.lang.Thread.run(Thread.java:748)

Reproduction line:

REPRODUCE WITH: ./gradlew :server:integTest -Dtests.seed=51682F94E41D8601 \ -Dtests.class=org.elasticsearch.index.seqno.RetentionLeaseSyncIT \ -Dtests.method="testRetentionLeasesSyncOnRecovery" -Dtests.security.manager=true \ -Dtests.locale=da -Dtests.timezone=Pacific/Gambier -Dcompiler.java=11 -Druntime.java=8 

I'l be muting this and I'm assigning Jason so that he gets the ping in case he wants to push a quick test fix

[elasticmachine]

Pinging @elastic/es-distributed

[jasontedor]

I have #38483 opened for this which addresses this by ensuring the index will not be relocated once we start creating retention leases.