[CI] ChangePolicyforIndexIT#testChangePolicyForIndex fails

[jimczi]

Two failures in 6x, however this does not reproduce for me:

org.junit.ComparisonFailure: expected:<[completed]> but was:<[hot]>
	at __randomizedtesting.SeedInfo.seed([E85178511074E054:9402C10DD3133A4]:0)
	at org.junit.Assert.assertEquals(Assert.java:115)
	at org.junit.Assert.assertEquals(Assert.java:144)
	at org.elasticsearch.xpack.indexlifecycle.ChangePolicyforIndexIT.assertStep(ChangePolicyforIndexIT.java:140)
	at org.elasticsearch.xpack.indexlifecycle.ChangePolicyforIndexIT.lambda$testChangePolicyForIndex$2(ChangePolicyforIndexIT.java:115)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:848)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:822)
	at org.elasticsearch.xpack.indexlifecycle.ChangePolicyforIndexIT.testChangePolicyForIndex(ChangePolicyforIndexIT.java:115)

The build was https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=opensuse/39/console:

REPRODUCE WITH: ./gradlew :x-pack:plugin:ilm:qa:multi-node:integTestRunner \
  -Dtests.seed=E85178511074E054 \
  -Dtests.class=org.elasticsearch.xpack.indexlifecycle.ChangePolicyforIndexIT \
  -Dtests.method="testChangePolicyForIndex" \
  -Dtests.security.manager=true \
  -Dtests.locale=en-SG \
  -Dtests.timezone=America/Nome \
  -Dcompiler.java=11 \
  -Druntime.java=8

Another run with the same issue:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=ubuntu&&virtual/36/console

[elasticmachine]

Pinging @elastic/es-core-infra

[talevy]

thanks @jimczi. I'll take a look at this

[talevy]

This looks like an issue of

2018-11-05T08:13:22,820][ERROR][o.e.x.i.IndexLifecycleRunner] [node-1] policy [policy_2] for index [test-000001] failed on step [{"phase":"hot","action":"rollover","name":"attempt_rollover"}]. Moving to ERROR step
org.elasticsearch.ResourceAlreadyExistsException: index [test-000002/Ts0s-CROTx65Th6tJDZ7pw] already exists

This has been addressed in #35168. I will keep an eye out for these issues today. If I do not see them on latest 6.x/master, then I will go ahead and close

[javanna]

Another failure, this one from master: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/149

[talevy]

thanks @javanna, this looks like another error up the change manifested by the change that meant to address the original issue.

I will mute the test and dig into this

relevant logs to debug:

[2018-11-06T01:05:38,088][ERROR][o.e.x.i.ExecuteStepsUpdateTask] [node-0] policy [policy_2] for index [test-000001] failed on cluster state step [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}]. Moving to ERROR step

java.lang.IllegalStateException: expected index [test-000001] with policy [policy_2] to have current step consistent with provided step key ({"phase":"hot","action":"complete","name":"complete"}) but it was {"phase":"hot","action":"rollover","name":"ERROR"}
	at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleRunner.maybeRunAsyncAction(IndexLifecycleRunner.java:163) ~[?:?]
	at org.elasticsearch.xpack.indexlifecycle.ExecuteStepsUpdateTask.clusterStateProcessed(ExecuteStepsUpdateTask.java:174) ~[?:?]

[talevy]

test muted in master: 2bf843e, and 6.x: c94d02a