[CI] testPrimaryRelocationWhileIndexing fails on master branch

[dnhatn]

testPrimaryRelocationWhileIndexing starts failing recently on the master branch.

ERROR   12.7s J0 | IndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing <<< FAILURES!
   > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2113, name=Thread-34, state=RUNNABLE, group=TGRP-IndexPrimaryRelocationIT]
   > 	at __randomizedtesting.SeedInfo.seed([2C153A0B57E3F301:72BAA35E02138F7A]:0)
   > Caused by: java.lang.AssertionError: expected:<DELETED> but was:<NOT_FOUND>
   > 	at __randomizedtesting.SeedInfo.seed([2C153A0B57E3F301]:0)
   > 	at org.elasticsearch.indices.recovery.IndexPrimaryRelocationIT$1.run(IndexPrimaryRelocationIT.java:64)

CI:

• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=opensuse/2340/console

• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=ubuntu/2340/console

• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=oraclelinux/2340/console

[elasticmachine]

Pinging @elastic/es-distributed

[dnhatn]

I have muted the test in 293f85c.

[dnhatn]

Previously we did not put an indexing to a version map if that map does not require safe access but removed the existing delete tombstone only if assertion enabled. In #29585, we removed the side-effect caused by assertion then this test started failing. This failure can be explained as follows:

• Step 1: Index a doc then delete that doc

• Step 2: The version map can switch to unsafe mode because of concurrent refreshes (implicitly called by flushes)

• Step 3: Index a document - the version map won't add this version value and won't prune the tombstone (previously it did)

• Step 4: Delete a document - this will return NOT_FOUND instead of DELETED because of the stale delete tombstone

This failure is actually fixed by #29619 in which we never leave stale delete tombstones.

/cc @bleskes