[ML] fixing testTwoJobsWithSameRandomizeSeedUseSameTrainingSet tests

[benwtrent]

This fixes the two test failures.

The shard failure seems to be due to the .ml-stats index being in the middle of being created.

See:

elasticsearch/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/action/TransportStartDataFrameAnalyticsAction.java

Lines 391 to 405 in d1e963e

	private void getProgress(DataFrameAnalyticsConfig config, ActionListener<List<PhaseProgress>> listener) {
	GetDataFrameAnalyticsStatsAction.Request getStatsRequest = new GetDataFrameAnalyticsStatsAction.Request(config.getId());
	executeAsyncWithOrigin(client, ML_ORIGIN, GetDataFrameAnalyticsStatsAction.INSTANCE, getStatsRequest, ActionListener.wrap(
	statsResponse -> {
	List<GetDataFrameAnalyticsStatsAction.Response.Stats> stats = statsResponse.getResponse().results();
	if (stats.isEmpty()) {
	// The job has been deleted in between
	listener.onFailure(ExceptionsHelper.missingDataFrameAnalytics(config.getId()));
	} else {
	listener.onResponse(stats.get(0).getProgress());
	}
	},
	listener::onFailure
	));
	}

We fail grabbing the progress on starting the second job.

This commit splits the creating/starting of the two jobs.

We still want to fail on shard failures like this as knowing the previous progress (if it exists) is vitally important.

closes #62064
closes #55807

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[droberts195]

This is OK as a fix for the test, but I don't like that we're having to change tests that are doing things that users could be doing (running two jobs together on a clean system).

It all goes back to the problem that searches treat indices that don't exist differently to indices that have just been created and are not yet ready to use (but which are obviously empty as they've only just been created). I will bump the thread about doing something about that.

[benwtrent]

It all goes back to the problem that searches treat indices that don't exist differently to indices that have just been created and are not yet ready to use

100% agree. I wish this "fix" was not necessary.