ClusterDisruptionIT#testAckedIndexing failures

[cbuescher]

On master: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+artifactory/329/console

Could not reproduce locally:

./gradlew :server:integTest --tests "org.elasticsearch.discovery.ClusterDisruptionIT.testAckedIndexing" \
  -Dtests.seed=E05E829B32C2E662 \
  -Dtests.security.manager=true \
  -Dtests.locale=en-SH \
  -Dtests.timezone=Europe/Oslo \
  -Dcompiler.java=12 \
  -Druntime.java=11

java.lang.AssertionError: AcknowledgedResponse failed - not acked
Expected: <true>
     but: was <false>
	at __randomizedtesting.SeedInfo.seed([E05E829B32C2E662:6A9F36686EBF0029]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked(ElasticsearchAssertions.java:112)
	at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked(ElasticsearchAssertions.java:116)
	at org.elasticsearch.test.TestCluster.wipeIndices(TestCluster.java:142)
	at org.elasticsearch.test.TestCluster.wipe(TestCluster.java:79)
	at org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:569)
	at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2170)
	at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)

[elasticmachine]

Pinging @elastic/es-distributed

[ywelsch]

The issue here is that disruptions are not always fully healed when the test ends, which leads to failures in clean-up actions. @DaveCTurner thoughts on how to ensure this?

[matriv]

Another failure encountered here: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+periodic/486/console

Reproduce with:

./gradlew :server:integTest --tests "org.elasticsearch.discovery.ClusterDisruptionIT.testAckedIndexing" \
  -Dtests.seed=D0453D059CF4F8AE \
  -Dtests.security.manager=true \
  -Dtests.locale=hr-HR \
  -Dtests.timezone=Pacific/Wake \
  -Dcompiler.java=12 \
  -Druntime.java=8

Couldn't reproduce locally.

The message is different though:

org.elasticsearch.discovery.ClusterDisruptionIT > testAckedIndexing FAILED
    java.lang.AssertionError: [test][2], node[kWppasG3RSKIRbBheOdOVA], [R], s[STARTED], a[id=SxfxSm_6Q3KWu6VDYMA0bQ] global checkpoint syncs mismatch
    Expected: <3L>
         but: was <4L>

Here are the complete logs in case they're lost from ci: https://gist.github.com/matriv/ee422e416aac35910194a7319a595ed6

[ywelsch]

@matriv I've figured out your failure. The issue was that the global checkpoint sync interval was not correctly reduced for testAckedIndexing. This is fixed in d49d9b5 /cc: @dnhatn

[dnhatn]

Thanks @ywelsch.

[henningandersen]

Backported this to 7.2 too, since these test-failures still show up here and this was clearly intended to go there too.
Failure:
https://scans.gradle.com/s/2r5a6flov3bfu/tests/kyv2y2z3r4v7m-f6hbue3yhtsqi