Failure in CoordinatorTests.testSingleNodeDiscoveryStabilisesEvenWhenDisrupted

[DaveCTurner]

CI Link

https://gradle-enterprise.elastic.co/s/yckwl5zjomfas

Repro line

./gradlew ':server:test' --tests "org.elasticsearch.cluster.coordination.CoordinatorTests.testSingleNodeDiscoveryStabilisesEvenWhenDisrupted" -Dtests.seed=949AB28501F8ED37:BBC20DD1A5D4EF81

Does it reproduce?

Yes

Applicable branches

main

Failure history

https://ci-stats.elastic.co/s/jenkins/goto/da1547f0-2ea6-11ed-8233-4bd40c8511bb - 3 similar-looking failures over the last 90 days

Failure excerpt

    java.lang.AssertionError: leaders
    Expected: not an empty collection
         but: was <[]>
        at __randomizedtesting.SeedInfo.seed([949AB28501F8ED37:BBC20DD1A5D4EF81]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.elasticsearch.cluster.coordination.AbstractCoordinatorTestCase$Cluster.getAnyLeader(AbstractCoordinatorTestCase.java:791)
        at org.elasticsearch.cluster.coordination.CoordinatorTests.testSingleNodeDiscoveryStabilisesEvenWhenDisrupted(CoordinatorTests.java:1939)


Suite: Test class org.elasticsearch.cluster.coordination.CoordinatorTests
  2> REPRODUCE WITH: ./gradlew ':server:test' --tests "org.elasticsearch.cluster.coordination.CoordinatorTests.testSingleNodeDiscoveryStabilisesEvenWhenDisrupted" -Dtests.seed=949AB28501F8ED37 -Dtests.locale=en-AU -Dtests.timezone=Europe/Vienna -Druntime.java=18
  2> java.lang.AssertionError: leaders
    Expected: not an empty collection
         but: was <[]>
        at __randomizedtesting.SeedInfo.seed([949AB28501F8ED37:BBC20DD1A5D4EF81]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.elasticsearch.cluster.coordination.AbstractCoordinatorTestCase$Cluster.getAnyLeader(AbstractCoordinatorTestCase.java:791)
        at org.elasticsearch.cluster.coordination.CoordinatorTests.testSingleNodeDiscoveryStabilisesEvenWhenDisrupted(CoordinatorTests.java:1939)
  2> NOTE: leaving temporary files on disk at: /home/davidturner/src/elasticsearch/server/build/testrun/test/temp/org.elasticsearch.cluster.coordination.CoordinatorTests_949AB28501F8ED37-002
  2> NOTE: test params are: codec=Asserting(Lucene94): {}, docValues:{}, maxPointsInLeafNode=1598, maxMBSortInHeap=6.485414969465074, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=en-AU, timezone=Europe/Vienna
  2> NOTE: Linux 5.4.0-125-generic amd64/Oracle Corporation 18.0.2.1 (64-bit)/cpus=8,threads=1,free=465272544,total=536870912
  2> NOTE: All tests run in this JVM: [CoordinatorTests]

Tests with failures:
 - org.elasticsearch.cluster.coordination.CoordinatorTests.testSingleNodeDiscoveryStabilisesEvenWhenDisrupted

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

Looks like the stabilisation timeout is just too short, because the following patch fixes this failure:

diff --git a/server/src/test/java/org/elasticsearch/cluster/coordination/CoordinatorTests.java b/server/src/test/java/org/elasticsearch/cluster/coordination/CoordinatorTests.java
index ca7cb106be6..49cd4a3a15b 100644
--- a/server/src/test/java/org/elasticsearch/cluster/coordination/CoordinatorTests.java
+++ b/server/src/test/java/org/elasticsearch/cluster/coordination/CoordinatorTests.java
@@ -1934,7 +1934,7 @@ public class CoordinatorTests extends AbstractCoordinatorTestCase {
             // detection and ongoing publications do not time out
             cluster.runFor(ELECTION_INITIAL_TIMEOUT_SETTING.get(Settings.EMPTY).millis() + delayVariabilityMillis
             // two round trips for pre-voting and voting
-                + 4 * delayVariabilityMillis
+                + 4 * delayVariabilityMillis + 300000
                 // and then the election update
                 + clusterStateUpdateDelay, "stabilising");

Ofc that's not the actual fix, we must work out for what this computation is not accounting.

[kingherc]

This is a hard. I started out to go back in history to figure out when this started appearing. It started appearing with https://github.com/elastic/elasticsearch/pull/86921/files , which removed the MockSinglePrioritizingExecutor in favor of some more java-native thread executor. With the old MockSinglePrioritizingExecutor, the specific test (and random seed) passes. However, in trying to understand the difference, I figured that MockSinglePrioritizingExecutor was flawed in multiple parts, e.g., it didn't emit a good string for the task to be executed, and also threw a KillWorkerError after executing a task, which created a different order of execution than the new java-native approach. I have the impression that during the KillWorkerError, the run() of the MockSinglePrioritizingExecutor was called again, and it might insert an additional "node join" tasks in the DeterministicTaskQueue, which might have been the reason why it could faster result in stabilization. In the end, due to all these differences, I ended up stopping the comparison with MockSinglePrioritizingExecutor. Will try to figure out with the code on main branch.

[kingherc]

@DaveCTurner I observe that simply adding another delayVariabilityMillis to the expression you mentioned passes the test. Would this be sufficient, or would we need to further understand the calculation?