[CI] NetworkDisruptionIT.testJobRelocation failing

[droberts195]

There have been a few recent failures of NetworkDisruptionIT.testJobRelocation:

• https://gradle-enterprise.elastic.co/s/u27wgfug6fd2c (master, macOS)
• https://gradle-enterprise.elastic.co/s/3aeahwzd7iwyi (master, Windows)
• https://gradle-enterprise.elastic.co/s/vbwc2iwn7vmzu (7.x, Windows)

In all cases the error is:

org.elasticsearch.xpack.ml.integration.NetworkDisruptionIT > testJobRelocation FAILED
    java.lang.AssertionError
        at org.junit.Assert.fail(Assert.java:86)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at org.junit.Assert.assertNotNull(Assert.java:712)
        at org.junit.Assert.assertNotNull(Assert.java:722)
        at org.elasticsearch.xpack.ml.support.BaseMlIntegTestCase.lambda$awaitJobOpenedAndAssigned$6(BaseMlIntegTestCase.java:400)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:891)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:864)
        at org.elasticsearch.xpack.ml.support.BaseMlIntegTestCase.awaitJobOpenedAndAssigned(BaseMlIntegTestCase.java:395)
        at org.elasticsearch.xpack.ml.integration.NetworkDisruptionIT.testJobRelocation(NetworkDisruptionIT.java:70)

    java.util.concurrent.ExecutionException: Failed to execute phase [query], 

        Caused by:
        Failed to execute phase [query], 

            Caused by:
            Failed to execute phase [query], Search rejected due to missing shards [[.ml-state][0]]. Consider using `allow_partial_search_results` setting to bypass this error.

This doesn't reproduce easily. It probably means there's somewhere in the test where we need to wait for the .ml-state index to have a usable replica prior to simulating the network disruption.

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[davidkyle]

The test failed due to an assertion waiting for the job to relocate with a 10 second timeout as we can see in the stack trace. The second error Failed to execute phase [query], occurs during the test teardown when the job is deleted and is likely caused be the job not relocating.

I suspect PersistentTasksClusterService::reassignPersistentTasks is not triggered during the 10 second window so it is not recognised that the job is unassigned and needs to be relocated. I enabled trace logging on PersistentTasksClusterService (master: 6070307, 7.x: 7d4118d) which will tell us if reassignPersistentTasks is called in the case of a failure

[davidkyle]

There has been another failure since trace logging was enabled which has shed some light and the problem is to do with the .ml-state index as suggested in the first comment.

https://gradle-enterprise.elastic.co/s/rtaxumxezptzw
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-windows-compatibility/os=windows-2012-r2/315

The master node is logging this message after the network disruption has been set

[2562-12-15T05:07:57,142][TRACE][o.e.p.PersistentTasksClusterService] [node_t3] reassigning 1 persistent tasks
[2562-12-15T05:07:57,143][TRACE][o.e.p.PersistentTasksClusterService] [node_t3] ignoring task job-relocation-job because assignment is the same node: [null], explanation: [Not opening job [relocation-job], because not all primary shards are active for the following indices [.ml-state]]

The job cannot relocate because ml-state is not available, this causes the failure. Later once the disruption is cleared the index becomes available.

[2562-12-15T05:08:07,302][INFO ][o.e.t.InternalTestCluster] [testJobRelocation] Clearing active scheme network disruption (disruption type: network disconnects, disrupted links: two partitions (partition 1: [node_t0] and partition 2: [node_t4, node_t3, node_t2, node_t1])), expected healing time 0s
[2562-12-15T05:08:07,302][INFO ][o.e.t.d.NetworkDisruption] [testJobRelocation] stop disrupting (disruption scheme: network disconnects, disrupted links: two partitions (partition 1: [node_t0] and partition 2: [node_t4, node_t3, node_t2, node_t1]))

...

[2562-12-15T05:08:08,060][INFO ][o.e.c.r.a.AllocationService] [node_t3] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.ml-state][0]]]).
[2562-12-15T05:08:08,131][INFO ][o.e.c.r.a.AllocationService] [node_t3] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.ml-state][0]]]).