[CI] TimeSeriesLifecycleActionsIT testWaitForSnapshot failure

[martijnvg]

Failure:

2> java.lang.AssertionError: 
    Expected: "completed"
         but: was "wait_for_snapshot"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.lambda$testWaitForSnapshot$23(TimeSeriesLifecycleActionsIT.java:344)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:879)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:852)
        at org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.testWaitForSnapshot(TimeSeriesLifecycleActionsIT.java:344)

Reproduce line:

./gradlew ':x-pack:plugin:ilm:qa:multi-node:integTestRunner' --tests "org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.testWaitForSnapshot" -Dtests.seed=EC28318704EF9660 -Dtests.security.manager=true -Dtests.locale=nb -Dtests.timezone=Pacific/Tarawa -Dcompiler.java=13

This didn't reproduce for me.

Build url: https://gradle-enterprise.elastic.co/s/uirpg2i73okrq

This test failed a few times already:
https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-30d,mode:quick,to:now))&_a=(columns:!(_source),index:e58bf320-7efd-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:'class:*TimeSeriesLifecycleActionsIT+AND+test:testWaitForSnapshot'),sort:!(time,desc))

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

[original-brownbear]

Muted in 170e100

[original-brownbear]

This looks like the same issue in testWaitForSnapshotSlmExecutedBefore https://gradle-enterprise.elastic.co/s/t65m4uewdrzlo

[mark-vieira]

Yeah, I'm seeing a number of failures for different tests in TimeSeriesLifecycleActionsIT starting around mid December.

Page takes a minute to load: https://gradle-enterprise.elastic.co/scans/tests?failures.failureClassification=non_verification&list.offset=0&list.size=50&list.sortColumn=startTime&list.sortOrder=desc&search.buildToolType=gradle&search.buildToolType=maven&search.startTimeMax=1578643199999&search.startTimeMin=1575187200000&tests.container=org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT&tests.sortField=FAILED&tests.unstableOnly&trends.section=overview&trends.timeResolution=day&viewer.tzOffset=-480

[nik9000]

This showed up with #50454

[mark-vieira]

Let's please give this some immediate attention. TimeSeriesLifecycleActionsIT is the most common test failure across CI in the past month excluding Windows packaging tests. That could just mean muting for now.