[CI] org.elasticsearch.xpack.ccr.action.ShardChangesTests testMissingOperations fails reproducably with seed

[colings86]

Build URL:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+matrix-java-periodic/ES_BUILD_JAVA=java11,ES_RUNTIME_JAVA=java11,nodes=virtual&&linux/44/console

Reproduce command (fails reproducably on 6.x but not on master) :

./gradlew :x-pack:plugin:ccr:test \
  -Dtests.seed=FAF7E61619663B98 \
  -Dtests.class=org.elasticsearch.xpack.ccr.action.ShardChangesTests \
  -Dtests.method="testMissingOperations" \
  -Dtests.security.manager=true \
  -Dtests.locale=en-VG \
  -Dtests.timezone=Europe/Istanbul \
  -Dcompiler.java=11 \
  -Druntime.java=11

It looks like this test is expecting operations to no longer be available and CCR to throw an exception but it doesn't get the exception thrown in this case

Stack trace:

FAILURE 4.09s | ShardChangesTests.testMissingOperations <<< FAILURES!
   > Throwable #1: junit.framework.AssertionFailedError: Expected exception ElasticsearchException but no exception was thrown
   >    at __randomizedtesting.SeedInfo.seed([FAF7E61619663B98:3B044284D8D8EB2]:0)
   >    at org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2697)
   >    at org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2682)
   >    at org.elasticsearch.xpack.ccr.action.ShardChangesTests.testMissingOperations(ShardChangesTests.java:114)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   >    at java.base/java.lang.Thread.run(Thread.java:834)

[elasticmachine]

Pinging @elastic/es-distributed

[colings86]

Muted in 6.x in 723f57a

[martijnvg]

@colings86 Thanks for raising this issue. I will look into this.

[martijnvg]

So I tried to reproduce this failure with reproduce command here and with just adding -Dtests.iters=128 and it does not reproduce for me. Not sure what the cause is.

[martijnvg]

I've increased the number of ops to make it less likely for all ops to remain as soft deleted docs: fba811f (which i think it the reason that the expected error wasn't thrown).

and also unmuted the test in 6.x branch.

[tvernum]

This just failed for me in a PR CI build (for master)

I can just rerun my build, but if you want the details for investigation purposes:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request/2469/console

REPRODUCE WITH: ./gradlew :x-pack:plugin:ccr:test -Dtests.seed=86846D2550B877E6 -Dtests.class=org.elasticsearch.xpack.ccr.action.ShardChangesTests -Dtests.method="testMissingOperations" -Dtests.security.manager=true -Dtests.locale=ru-RU -Dtests.timezone=Antarctica/Palmer -Dcompiler.java=11 -Druntime.java=8

03:31:21 FAILURE 5.86s J3 | ShardChangesTests.testMissingOperations <<< FAILURES!
03:31:21    > Throwable #1: junit.framework.AssertionFailedError: Expected exception ElasticsearchException but no exception was thrown
03:31:21    > 	at __randomizedtesting.SeedInfo.seed([86846D2550B877E6:7FC3CF1B0453C2CC]:0)
03:31:21    > 	at org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2684)
03:31:21    > 	at org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2669)
03:31:21    > 	at org.elasticsearch.xpack.ccr.action.ShardChangesTests.testMissingOperations(ShardChangesTests.java:114)
03:31:21    > 	at java.lang.Thread.run(Thread.java:748)

[martijnvg]

@tvernum Thanks for sharing this. I think that the force merge did not really merge anything, and therefor kept the soft deletes. I pushed a change that should avoid such scenarios: 0487181

[martijnvg]

Closing. I've not seen this test failing after the test was adjusted.