Watcher never fuly starts in rolling upgrade tests #58282
Description
Build scan:
scan
Repro line:
./gradlew -p x-pack/qa/rolling-upgrade check
Reproduces locally?:
Not for me.
Applicable branches: Looks like just master.
Failure history:
spiked around June 15th.
Failure excerpt:
REPRODUCE WITH: ./gradlew ':x-pack:qa:rolling-upgrade:v7.9.0#twoThirdsUpgradedTest' --tests "org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=mixed_cluster/30_ml_jobs_crud/Test job with pre 6.4 rules}" -Dtests.seed=4AEEA945B9AA76DA -Dtests.security.manager=true -Dtests.locale=es-AR -Dtests.timezone=CNT -Druntime.java=15 -Dtests.rest.suite=mixed_cluster
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT > test {p0=mixed_cluster/30_ml_jobs_crud/Test job with pre 6.4 rules} FAILED
java.lang.AssertionError:
Expected: a value equal to or greater than <3>
but: <2> was less than <3>
at __randomizedtesting.SeedInfo.seed([4AEEA945B9AA76DA:C2BA969F17561B22]:0)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT.lambda$waitForWatcher$0(UpgradeClusterClientYamlTestSuiteIT.java:50)
at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:892)
at org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT.waitForWatcher(UpgradeClusterClientYamlTestSuiteIT.java:46)
:x-pack:qa:full-cluster-restart:v7.9.0#upgradedClusterTest