IndexFollowingIT#testFollowIndex can fail with assertion error.

[jtibshirani]

I wasn't able to reproduce the error locally. It looks like this failure has popped up 2-3 times in CI over the past month. It may be related to #38949, feel free to close this in favor of that one if you'd prefer to track work there.

---

Link to the build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/3039/console

Command to reproduce:

./gradlew :x-pack:plugin:ccr:internalClusterTest --tests "org.elasticsearch.xpack.ccr.IndexFollowingIT.testFollowIndex" \
  -Dtests.seed=899F6DBC812AC342 \
  -Dtests.security.manager=true \
  -Dtests.locale=mer-KE \
  -Dtests.timezone=America/Fortaleza \
  -Dcompiler.java=12 \
  -Druntime.java=11

Relevant excerpt from the logs:

org.elasticsearch.xpack.ccr.IndexFollowingIT > testFollowIndex FAILED
    java.lang.AssertionError: 
    Expected: <0>
         but: was <1>
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.xpack.CcrIntegTestCase.lambda$ensureNoCcrTasks$3(CcrIntegTestCase.java:418)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:862)
        at org.elasticsearch.xpack.CcrIntegTestCase.ensureNoCcrTasks(CcrIntegTestCase.java:399)
        at org.elasticsearch.xpack.CcrIntegTestCase.pauseFollow(CcrIntegTestCase.java:395)
        at org.elasticsearch.xpack.ccr.IndexFollowingIT.testFollowIndex(IndexFollowingIT.java:191)

    java.lang.AssertionError: not all translog generations have been released

    com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=277, name=elasticsearch[follower1][generic][T#4], state=RUNNABLE, group=TGRP-IndexFollowingIT]

        Caused by:
        java.lang.AssertionError: invalid global checkpoint[5596] source_meta_data [{local_checkpoint=5596, max_unsafe_auto_id_timestamp=-1, min_retained_seq_no=0, translog_uuid=oPl81yVBToWbh_M-LBrEqA, history_uuid=rudbNf_KSZuvLto5xdE5nQ, translog_generation=1, max_seq_no=5889}]

[elasticmachine]

Pinging @elastic/es-distributed