TimeSeriesLifecycleActionsIT muliple tests fail

[jkakavas]

we have 3 open issues https://github.com/elastic/elasticsearch/search?q=TimeSeriesLifecycleActionsIT&type=Issues so I thnk we can consolidate all these here and I'll mute the whole class

https://gradle-enterprise.elastic.co/s/p6yto4rhlkr2a

org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT > testRetryFailedDeleteAction FAILED
    java.lang.AssertionError: 
    Expected: "delete"
         but: was null
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.lambda$testRetryFailedDeleteAction$8(TimeSeriesLifecycleActionsIT.java:222)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:887)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:860)
        at org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.testRetryFailedDeleteAction(TimeSeriesLifecycleActionsIT.java:222)

    org.elasticsearch.client.ResponseException: method [DELETE], host [http://127.0.0.1:51980], URI [*?expand_wildcards=open%2Cclosed%2Chidden], status line [HTTP/1.1 403 Forbidden]
    {"error":{"root_cause":[{"type":"cluster_block_exception","reason":"index [index-wdjrnbsiia] blocked by: [FORBIDDEN/5/index read-only (api)];"}],"type":"cluster_block_exception","reason":"index [index-wdjrnbsiia] blocked by: [FORBIDDEN/5/index read-only (api)];"},"status":403}
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:283)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:261)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
        at org.elasticsearch.test.rest.ESRestTestCase.wipeAllIndices(ESRestTestCase.java:577)
        at org.elasticsearch.test.rest.ESRestTestCase.wipeCluster(ESRestTestCase.java:525)
        at org.elasticsearch.test.rest.ESRestTestCase.cleanUpCluster(ESRestTestCase.java:293)
        at jdk.internal.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:834)
REPRODUCE WITH: ./gradlew ':x-pack:plugin:ilm:qa:multi-node:integTestRunner' --tests "org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.testRetryFailedDeleteAction" -Dtests.seed=CC45FDA1AE932803 -Dtests.security.manager=true -Dtests.locale=mk-MK -Dtests.timezone=Mexico/BajaNorte -Dcompiler.java=13

as a note testRetryFailedDeleteAction seems to started failing when #52494 was merged ( see here )

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

[dakrone]

@jkakavas this class has over 45 tests, I don't think we should mute the entire class because 3-4 tests fail, that can lead to a regression for the stable tests in this class. There is an issue already open #53488 to refactor it into multiple classes, but I think muting the entire class is a bad idea here.

[jkakavas]

fair enough, I'll revert this.

[dakrone]

Please do mute the tests that did fail, I agree wholeheartedly for doing that, I just don't want to skip all the passing tests in this class.

[jkakavas]

I just want to note that there are many tests that fail in this suite, not just 2,3 : https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-30d,mode:quick,to:now))&_a=(columns:!(test),index:e58bf320-7efd-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:TimeSeriesLifecycleActionsIT),sort:!(time,desc)) and it's hard for me ( as in random ES person on test triage ) to know if the historical failures of the last month are fixed in the meantime, or just never muted etc.

We can take a failure at a time if you think this is better, I don't have a reason to push back, I just wanted to make sure the extent of the failures is known