[CI] TransformSurvivesUpgradeIT testTransformRollingUpgrade failing

[hendrikmuhs]

probably a regression of #80013

Build scan:
https://gradle-enterprise.elastic.co/s/toqidjvpswti4/tests/:x-pack:qa:rolling-upgrade:v7.11.2%23upgradedClusterTest/org.elasticsearch.upgrades.TransformSurvivesUpgradeIT/testTransformRollingUpgrade

Reproduction line:
./gradlew ':x-pack:qa:rolling-upgrade:v7.11.2#upgradedClusterTest' -Dtests.class="org.elasticsearch.upgrades.TransformSurvivesUpgradeIT" -Dtests.method="testTransformRollingUpgrade" -Dtests.seed=11151805F4A613BA -Dtests.bwc=true -Dtests.locale=und -Dtests.timezone=America/Indiana/Knox -Druntime.java=8

Applicable branches:
7.16

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.upgrades.TransformSurvivesUpgradeIT&tests.test=testTransformRollingUpgrade

Failure excerpt:

java.lang.AssertionError: 
Expected: a value greater than <7>
     but: <7> was equal to <7>

  at __randomizedtesting.SeedInfo.seed([11151805F4A613BA:BE682DBC5FE0C07E]:0)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
  at org.junit.Assert.assertThat(Assert.java:956)
  at org.junit.Assert.assertThat(Assert.java:923)
  at org.elasticsearch.upgrades.TransformSurvivesUpgradeIT.lambda$verifyContinuousTransformHandlesData$8(TransformSurvivesUpgradeIT.java:248)
  at org.elasticsearch.upgrades.TransformSurvivesUpgradeIT.lambda$awaitWrittenIndexerState$9(TransformSurvivesUpgradeIT.java:301)
  at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1123)
  at org.elasticsearch.upgrades.TransformSurvivesUpgradeIT.awaitWrittenIndexerState(TransformSurvivesUpgradeIT.java:293)
  at org.elasticsearch.upgrades.TransformSurvivesUpgradeIT.verifyContinuousTransformHandlesData(TransformSurvivesUpgradeIT.java:243)
  at org.elasticsearch.upgrades.TransformSurvivesUpgradeIT.testTransformRollingUpgrade(TransformSurvivesUpgradeIT.java:153)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  at java.lang.Thread.run(Thread.java:748)

[elasticmachine]

Pinging @elastic/ml-core (Team:ML)

[hendrikmuhs]

There is still 1 corner case (https://gradle-enterprise.elastic.co/s/d5qblitdhiy6g):

A transform that is running during upgrade but hasn't written a state document yet. In case upgrade runs before it can write a state doc into the new index, the upgrader "wins" this battle. Because the task uses optimistic concurrency control it stumbles upon the state being written while it runs.

This is a known issue which hit us in other places as well.

Severity: This is a rather unlikely problem in production but can be considered as a typical automation bug in CI, because you must hit the tiny time window and start the upgrade right after the last node has upgraded. Note that the transform already runs in the mixed cluster, so it must not write state for some time to make this happen.

If this issue creates too much noise, please mute it, but only for 7.16, this does not happen in 8.x branches.

possible solutions:
A: don't upgrade the state document in upgrade for running tasks. However this has the problem that we want to cleanup old indices, but deleting the old indices when state hasn't been persisted is not a good idea
B: message a running task to persist state instead of writing the state
C: improve version conflict handling in persistState
D: update state and checkpoint on task relocation if they are in an old format

[gwbrown]

Today, this failed on nearly every BWC test upgrading to 7.16.0 from 7.4.0 through 7.12.1. Links to sample scans for when the matrix ages off CI:

• 7.16.0/7.4.0
• 7.16.0/7.6.0
• 7.16.0/7.5.2
• 7.16.0/7.4.2