[CI] RecoveryIT.testRecoveryWithConcurrentIndexing fails on master expecting more documents

[dakrone]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/931/console

FAILURE 2.01s | RecoveryIT.testRecoveryWithConcurrentIndexing <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: 
   > Expected: <60>
   >      but: was <110>
   > 	at __randomizedtesting.SeedInfo.seed([65E63137F3BA894E:E5B7F564711E5784]:0)
   > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 	at org.elasticsearch.upgrades.RecoveryIT.assertCount(RecoveryIT.java:204)
   > 	at org.elasticsearch.upgrades.RecoveryIT.testRecoveryWithConcurrentIndexing(RecoveryIT.java:192)
   > 	at java.lang.Thread.run(Thread.java:748)

Expecting that the first node have 60 documents, instead has 110:

            case UPGRADED:
                updateIndexSetting(index, Settings.builder().put(INDEX_ROUTING_ALLOCATION_ENABLE_SETTING.getKey(), (String)null));
                asyncIndexDocs(index, 60, 50).get();
                ensureGreen(index);
                assertOK(client().performRequest("POST", index + "/_refresh"));
                assertCount(index, "_only_nodes:" + nodes.get(0), 60); // <-- failure here
                assertCount(index, "_only_nodes:" + nodes.get(1), 60);
                break;

Reproduces for me with:

gradle :qa:rolling-upgrade:v6.2.0-SNAPSHOT#upgradedClusterTestRunner -Dtests.seed=65E63137F3BA894E -Dtests.class=org.elasticsearch.upgrades.RecoveryIT -Dtests.method="testRecoveryWithConcurrentIndexing" -Dtests.security.manager=true -Dtests.locale=sl -Dtests.timezone=SST

[bleskes]

This is basically a git cherry pick fail. I fixed it only to lose it when I was cleaning up the commits... Thx Lee!

[imotov]

@bleskes it looks like it failed again with exactly the same error message:

java.lang.AssertionError: 
Expected: <110>
     but: was <60>
	at __randomizedtesting.SeedInfo.seed([887030CB7438355C:821F498F69CEB96]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.upgrades.RecoveryIT.assertCount(RecoveryIT.java:168)
	at org.elasticsearch.upgrades.RecoveryIT.testRecoveryWithConcurrentIndexing(RecoveryIT.java:156)

REPRODUCE WITH: ./gradlew :qa:rolling-upgrade:v5.6.12-SNAPSHOT#upgradedClusterTestRunner \
  -Dtests.seed=887030CB7438355C \
  -Dtests.class=org.elasticsearch.upgrades.RecoveryIT \
  -Dtests.method="testRecoveryWithConcurrentIndexing" \
  -Dtests.security.manager=true \
  -Dtests.locale=fr-FR \
  -Dtests.timezone=CNT \
  -Dcompiler.java=10 \
  -Druntime.java=8FIPS \
  -Djavax.net.ssl.keyStorePassword=password \
  -Djavax.net.ssl.trustStorePassword=password

CI: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java8fips,nodes=virtual&&linux/297/console

Should this be cherry-picked somewhere else as well?

[dnhatn]

I think this test fails because we rollback a replica to the global checkpoint(#33473) but do not restore its history on promotion. This should be fixed by #33616.

[dnhatn]

Fixed by #33616.

[dnhatn]

This fails on 6.x: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/46/console

FAILURE 1.54s | RecoveryIT.testRecoveryWithConcurrentIndexing <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: 
   > Expected: <110>
   >      but: was <60>
   > 	at __randomizedtesting.SeedInfo.seed([AD22BB7534737F0E:2D737F26B6D7A1C4]:0)
   > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 	at org.elasticsearch.upgrades.RecoveryIT.assertCount(RecoveryIT.java:168)
   > 	at org.elasticsearch.upgrades.RecoveryIT.testRecoveryWithConcurrentIndexing(RecoveryIT.java:156)

Log: testRecoveryWithConcurrentIndexing.txt.zip

[elasticmachine]

Pinging @elastic/es-distributed