[CI] Index synced flush rest test sees 409 Conflict #29162
Description
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.2+intake/162/consoleText
1> [2018-03-19T19:10:22,030][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test {p0=indices.flush/10_basic/Index synced flush rest test}]: before test
1> [2018-03-19T19:10:22,156][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] Stash dump on test failure [{
1> "stash" : {
1> "body" : {
1> "_shards" : {
1> "total" : 5,
1> "successful" : 3,
1> "failed" : 2
1> },
1> "testing" : {
1> "total" : 5,
1> "successful" : 3,
1> "failed" : 2,
1> "failures" : [
1> {
1> "shard" : 2,
1> "reason" : "[node-1][127.0.0.1:38733][internal:indices/flush/synced/sync] disconnected",
1> "routing" : {
1> "state" : "STARTED",
1> "primary" : true,
1> "node" : "CDF_2XQuR3apYhenQdnl7w",
1> "relocating_node" : null,
1> "shard" : 2,
1> "index" : "testing",
1> "allocation_id" : {
1> "id" : "lWfMa2hRRBuwosyqnQ6TmQ"
1> }
1> }
1> },
1> {
1> "shard" : 1,
1> "reason" : "[node-0][127.0.0.1:46759][internal:indices/flush/synced/sync] disconnected",
1> "routing" : {
1> "state" : "STARTED",
1> "primary" : true,
1> "node" : "2pbkQL89Q2Wf0hNye6uxyQ",
1> "relocating_node" : null,
1> "shard" : 1,
1> "index" : "testing",
1> "allocation_id" : {
1> "id" : "jQYe2kcRR_2uduMr8FpK9A"
1> }
1> }
1> }
1> ]
1> }
1> }
1> }
1> }]
1> [2018-03-19T19:10:22,241][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test {p0=indices.flush/10_basic/Index synced flush rest test}]: after test
FAILURE 0.22s | MixedClusterClientYamlTestSuiteIT.test {p0=indices.flush/10_basic/Index synced flush rest test} <<< FAILURES!
> Throwable #1: java.lang.AssertionError: Failure at [indices.flush/10_basic:14]: expected [2xx] status code but api [indices.flush_synced] returned [409 Conflict] [{"_shards":{"total":5,"successful":3,"failed":2},"testing":{"total":5,"successful":3,"failed":2,"failures":[{"shard":2,"reason":"[node-1][127.0.0.1:38733][internal:indices/flush/synced/sync] disconnected","routing":{"state":"STARTED","primary":true,"node":"CDF_2XQuR3apYhenQdnl7w","relocating_node":null,"shard":2,"index":"testing","allocation_id":{"id":"lWfMa2hRRBuwosyqnQ6TmQ"}}},{"shard":1,"reason":"[node-0][127.0.0.1:46759][internal:indices/flush/synced/sync] disconnected","routing":{"state":"STARTED","primary":true,"node":"2pbkQL89Q2Wf0hNye6uxyQ","relocating_node":null,"shard":1,"index":"testing","allocation_id":{"id":"jQYe2kcRR_2uduMr8FpK9A"}}}]}}]
> at __randomizedtesting.SeedInfo.seed([8FC9D8CFC41D4733:79DE7156AE12ACB]:0)
> at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:343)
> at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:320)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.AssertionError: expected [2xx] status code but api [indices.flush_synced] returned [409 Conflict] [{"_shards":{"total":5,"successful":3,"failed":2},"testing":{"total":5,"successful":3,"failed":2,"failures":[{"shard":2,"reason":"[node-1][127.0.0.1:38733][internal:indices/flush/synced/sync] disconnected","routing":{"state":"STARTED","primary":true,"node":"CDF_2XQuR3apYhenQdnl7w","relocating_node":null,"shard":2,"index":"testing","allocation_id":{"id":"lWfMa2hRRBuwosyqnQ6TmQ"}}},{"shard":1,"reason":"[node-0][127.0.0.1:46759][internal:indices/flush/synced/sync] disconnected","routing":{"state":"STARTED","primary":true,"node":"2pbkQL89Q2Wf0hNye6uxyQ","relocating_node":null,"shard":1,"index":"testing","allocation_id":{"id":"jQYe2kcRR_2uduMr8FpK9A"}}}]}}]
> at org.elasticsearch.test.rest.yaml.section.DoSection.execute(DoSection.java:238)
> at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:336)
> ... 37 more
REPRODUCE WITH: ./gradlew :qa:mixed-cluster:v5.6.9-SNAPSHOT#mixedClusterTestRunner -Dtests.seed=8FC9D8CFC41D4733 -Dtests.class=org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT -Dtests.method="test {p0=indices.flush/10_basic/Index synced flush rest test}" -Dtests.security.manager=true -Dtests.locale=en-PH -Dtests.timezone=Asia/Srednekolymsk
The only change on 6.2 was a doc fix (#29116) but #29103 (on the 5.6 branch) looks like it's a more likely culprit. @dnhatn could you take a look?