ClassificationIT testTwoJobsWithSameRandomizeSeedUseSameTrainingSet failures

[davidkyle]

Build scan:
https://gradle-enterprise.elastic.co/s/alrafnmsinb3k/console-log?task=:x-pack:plugin:ml:qa:native-multi-node-tests:javaRestTest

Repro line:

./gradlew ':x-pack:plugin:ml:qa:native-multi-node-tests:javaRestTest' --tests "org.elasticsearch.xpack.ml.integration.ClassificationIT.testTwoJobsWithSameRandomizeSeedUseSameTrainingSet" \
  -Dtests.seed=2E25F8B5498350DB \
  -Dtests.security.manager=true \
  -Dtests.locale=th-TH-u-nu-thai-x-lvariant-TH \
  -Dtests.timezone=Europe/Madrid \
  -Druntime.java=11

Reproduces locally?:
No

Applicable branches:
master

Failure history:
Common

Failure excerpt:

org.elasticsearch.ElasticsearchException: all shards failed
	at __randomizedtesting.SeedInfo.seed([2E25F8B5498350DB:6B2EF9D391841721]:0)
	at org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.serverError(ExceptionsHelper.java:55)
	at org.elasticsearch.xpack.ml.action.TransportGetDataFrameAnalyticsStatsAction.lambda$searchStats$8(TransportGetDataFrameAnalyticsStatsAction.java:233)

From the log the interesting message is

[2020-09-07T11:20:50,053][ERROR][o.e.x.m.a.TransportGetDataFrameAnalyticsStatsAction] [javaRestTest-0] [classification_two_jobs_with_same_randomize_seed_2] Item failure encountered during multi search for request [indices=[.ml-state-write, .ml-state-000001], source={"size":1,"query":{"ids":{"values":["data_frame_analytics-classification_two_jobs_with_same_randomize_seed_2-progress"],"boost":1.0}}}]: all shards failed
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed

indices=[.ml-state-write, .ml-state-000001]

Logs attached
logs.zip

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[davidkyle]

I muted the test. The error looks like it could occur anywhere but this test seems to provoke it more than the others.

[benwtrent]

Digging into the logs,

I don't see .ml-state-write alias being created during the time frame of this test...

Other interesting info:

The node that threw the failure was elected as master.

That same node later had these entries:

[2020-09-07T11:28:28,253][INFO ][o.e.c.m.MetadataCreateIndexService] [javaRestTest-0] [.ml-state-write] creating index, cause [auto(bulk api)], templates [.ml-state], shards [1]/[1]

I thought we had put code everywhere that disallowed this type of action. But obviously not.

[benwtrent]

See: #62784

The creation of a concrete .ml-state-write index should be resolved.

If this test continues to fail, at least we know it is not that.

[davidkyle]

@benwtrent is this the failure you were looking for?
https://gradle-enterprise.elastic.co/s/peemwo6c3adqm

[benwtrent]

@davidkyle yep!