[CI] SnapshotDisruptionIT. testDisruptionOnSnapshotInitialization fails unreproducibly

[spinscale]

See https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/837/console

Ran this with hundred iterations on Linux and once on osx, does not reproduce. Reproduction line is

 ./gradlew :server:integTest \
  -Dtests.seed=642A19B3B0D37154 \
  -Dtests.class=org.elasticsearch.discovery.SnapshotDisruptionIT \
  -Dtests.method="testDisruptionOnSnapshotInitialization" \
  -Dtests.security.manager=true \
  -Dtests.locale=hr-HR \
  -Dtests.timezone=Europe/Tirane \
  -Dcompiler.java=11 \
  -Druntime.java=8

I saw another test failure in the last days. Log excerpt

  1> [2018-12-18T14:58:52,986][INFO ][o.e.d.SnapshotDisruptionIT] [testDisruptionOnSnapshotInitialization] after test
FAILURE 4.73s J1 | SnapshotDisruptionIT.testDisruptionOnSnapshotInitialization <<< FAILURES!
   > Throwable #1: java.lang.AssertionError:
   > Expected: an instance of org.elasticsearch.discovery.MasterNotDiscoveredException
   >      but: <RemoteTransportException[[node_tm1][127.0.0.1:41899][cluster:admin/snapshot/create]]; nested: ConcurrentSnaps
hotExecutionException[[test-repo:test-snap-2] cannot snapshot while a snapshot deletion is in-progress];> is a org.elasticsea
rch.transport.RemoteTransportException
   >    at __randomizedtesting.SeedInfo.seed([642A19B3B0D37154:4EF53F6396CB828B]:0)
   >    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >    at org.elasticsearch.discovery.SnapshotDisruptionIT.testDisruptionOnSnapshotInitialization(SnapshotDisruptionIT.java:
164)
   >    at java.lang.Thread.run(Thread.java:748)

Direct log https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/837/consoleText

[elasticmachine]

Pinging @elastic/es-distributed

[original-brownbear]

Fixed incoming in #36801 I think :)

[javanna]

Another occurrence: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=java8fips,nodes=virtual&&linux/145/console

[javanna]

and one more: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+g1gc/218/console

[original-brownbear]

Reproduced this again by putting a 5s sleep into the task in org.elasticsearch.snapshots.SnapshotsService#deleteSnapshotFromRepository. The same reproducer was fixed before ... seems something changed meanwhile. Looking into it.

Update:

Nothing has changed, it's just this seed (C79E004B4B9ADD0C) that is failing if I artificially slow down the deletion of the snapshot.