[CI] SLMSnapshotBlockingIntegTests.testBasicPartialRetention failed

[droberts195]

SLMSnapshotBlockingIntegTests.testBasicPartialRetention failed in https://gradle-enterprise.elastic.co/s/shm65wzp2dh54

The error is:

org.elasticsearch.xpack.slm.SLMSnapshotBlockingIntegTests > testBasicPartialRetention FAILED
org.elasticsearch.snapshots.SnapshotMissingException: [my-repo:snap-hfs4h9aptqkjnchyhkltla] is missing

This did not reproduce locally using:

./gradlew ':x-pack:plugin:ilm:test' --tests "org.elasticsearch.xpack.slm.SLMSnapshotBlockingIntegTests.testBasicPartialRetention" \
  -Dtests.seed=EAF550063D3EF24D \
  -Dtests.security.manager=true \
  -Dtests.locale=nn \
  -Dtests.timezone=America/Kentucky/Louisville \
  -Dcompiler.java=12 \
  -Druntime.java=11

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

[droberts195]

It's a different test and a different branch (7.x) but SLMSnapshotBlockingIntegTests.testRetentionWhileSnapshotInProgress also failed in https://gradle-enterprise.elastic.co/s/pmnulxslshgs4 with the same error message:

[2019-10-16T13:28:40,809][INFO ][o.e.c.r.a.AllocationService] [node_tm1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][0]]]).
[2019-10-16T13:28:40,811][INFO ][o.e.x.s.SnapshotLifecycleTask] [node_tm1] snapshot lifecycle policy [slm-policy] issuing create snapshot [snap-x-u632e9qjmkrbe08kz_cq]
[2019-10-16T13:28:40,815][INFO ][o.e.x.s.SLMSnapshotBlockingIntegTests] [testRetentionWhileSnapshotInProgress] --> kicked off snapshot snap-x-u632e9qjmkrbe08kz_cq
[2019-10-16T13:28:40,823][ERROR][o.e.x.s.SLMSnapshotBlockingIntegTests] [testRetentionWhileSnapshotInProgress] expected a snapshot but it was missing
org.elasticsearch.snapshots.SnapshotMissingException: [repo-id:snap-x-u632e9qjmkrbe08kz_cq] is missing
	at org.elasticsearch.action.admin.cluster.snapshots.status.TransportSnapshotsStatusAction.buildResponse(TransportSnapshotsStatusAction.java:214) ~[elasticsearch-7.5.0-SNAPSHOT.jar:7.5.0-SNAPSHOT]
	at org.elasticsearch.action.admin.cluster.snapshots.status.TransportSnapshotsStatusAction.masterOperation(TransportSnapshotsStatusAction.java:100) ~[elasticsearch-7.5.0-SNAPSHOT.jar:7.5.0-SNAPSHOT]
	at org.elasticsearch.action.admin.cluster.snapshots.status.TransportSnapshotsStatusAction.masterOperation(TransportSnapshotsStatusAction.java:61) ~[elasticsearch-7.5.0-SNAPSHOT.jar:7.5.0-SNAPSHOT]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:98) ~[elasticsearch-7.5.0-SNAPSHOT.jar:7.5.0-SNAPSHOT]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:169) ~[elasticsearch-7.5.0-SNAPSHOT.jar:7.5.0-SNAPSHOT]
	at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) ~[elasticsearch-7.5.0-SNAPSHOT.jar:7.5.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) ~[elasticsearch-7.5.0-SNAPSHOT.jar:7.5.0-SNAPSHOT]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.5.0-SNAPSHOT.jar:7.5.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_221]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_221]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_221]

The repro command for that one is:

./gradlew ':x-pack:plugin:ilm:test' --tests "org.elasticsearch.xpack.slm.SLMSnapshotBlockingIntegTests.testRetentionWhileSnapshotInProgress" \
  -Dtests.seed=278E4966E79E2EA5 \
  -Dtests.security.manager=true \
  -Dtests.locale=fi-FI \
  -Dtests.timezone=MET \
  -Dcompiler.java=12 \
  -Druntime.java=8

This doesn't reproduce locally on the 7.x branch.

[gwbrown]

I don't think these are the same issue - the latter does have the same exception in the logs, but that isn't the cause of the failure. In the testRetentionWhileSnapshotInProgress that message is logged each time an assertBusy fails, but the test eventually progresses from that once the snapshot exists, then fails on a different assertion. I'll open a new issue for this shortly.

In the testBasicPartialRetention failure, the failure is due to a snapshot being missing that we have previously verified exists, and, per the logs, has not been deleted, so I am very confused.

[gwbrown]

I've opened #48159 for the testRetentionWhileSnapshotInProgress failures so that this issue can stay focused on the testBasicPartialRetention failure.

[romseygeek]

This failed again on 7.5: https://gradle-enterprise.elastic.co/s/lsfer4ioyhnag/console-log?task=:x-pack:plugin:ilm:test

[original-brownbear]

Phew :) -> tracked this down in #48329