index sorting yaml test fails against multiple nodes

[javanna]

Link to the failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-intake/1160 .

Despite setting the index to be sorted on the rank field, it seems like the results are not properly sorted.

> Throwable #1: java.lang.AssertionError: Failure at [indices.sort/10_basic:72]: hits.hits.0._id didn't match expected value:
   >                hits.hits.0._id: expected [2] but was [3]
   > 	at __randomizedtesting.SeedInfo.seed([E5B9D8E6C696CADE:6DEDE73C686AA726]:0)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:346)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:326)
   > 	at java.lang.Thread.run(Thread.java:748)
   > Caused by: java.lang.AssertionError: hits.hits.0._id didn't match expected value:
   >                hits.hits.0._id: expected [2] but was [3]
   > 	at org.elasticsearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:87)
   > 	at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:76)
   > 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:342)
   > 	... 37 more

[mayya-sharipova]

The index sorting test still fails elastic/elasticsearch 6.x 2018-02-08T16:40:51.597Z:

Failure at [indices.sort/10_basic:143]: hits.hits.6._id didn't match expected value:
hits.hits.6._id: expected [8] but was [5]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+oracle-java9-periodic/69/console
REPRODUCE WITH:

./gradlew :qa:smoke-test-multinode:integTestRunner \
  -Dtests.seed=FBD2C92E4D563B0 \
  -Dtests.class=org.elasticsearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT \
  -Dtests.method="test {yaml=indices.sort/10_basic/Index Sort}" \
  -Dtests.security.manager=true \
  -Dtests.locale=en-RW \
  -Dtests.timezone=America/Indiana/Indianapolis

@jimczi I have noticed that that you reverted back : number_of_replicas: 1 here. Is there a reason to have it to be equal to 1 and not 0? It seems that the test fails because of number_of_replicas is 1

[javanna]

@mayya-sharipova were you able to repro with the seed? I've seen a similar failure also on master (https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+oracle-java10-periodic/61) yet I can not repro on any branch with any seed.

[mayya-sharipova]

@javanna I was not able to reproduce it on my local machine. I guess the test fails when replica lags behind the main shard in updating docs.

[romseygeek]

I think what is happening here is that the indexing and then force-merge on the master are not necessarily being replayed in exactly the same order on the replica, and so instead of ending up with a single sorted segment, we have two segments, the second of which only contains document 8. Although there's a refresh and two searches in between the last index and the force-merge, so how the reordering is happening isn't obvious.

We can stick a few extra flush or refresh commands in there to ensure that everything is in the correct order. Or as @mayya-sharipova suggested, we could just make the test a 0-replica test, as it doesn't really add anything to have a replica here?