[ML][Data frame] fixing failure state transitions and race condition (#45627)

[benwtrent]

There is a small window for a race condition while we are flagging a task as failed.

Here are the steps where the race condition occurs:

1. A failure occurs
2. Before AsyncTwoPhaseIndexer calls the onFailure handler it does the following:
   a. finishAndSetState() which sets the IndexerState to STARTED
   b. doSaveState(...) which attempts to save the current state of the indexer
3. Another trigger is fired BEFORE onFailure can fire, but AFTER finishAndSetState() occurs.

The trick here is that we will eventually set the indexer to failed, but possibly not before another trigger had the opportunity to fire. This could obviously cause some weird state interactions. To combat this, I have put in some predicates to verify the state before taking actions. This is so if state is indeed marked failed, the "second trigger" stops ASAP.

Additionally, I move the task state checks INTO the start and stop methods, which will now require a force parameter. start, stop, trigger and markAsFailed are all synchronized. This should gives us some guarantees that one will not switch states out from underneath another.

I also flag the task as failed BEFORE we successfully write it to cluster state, this is to allow us to make the task fail more quickly. But, this does add the behavior where the task is "failed" but the cluster state does not indicate as much. Adding the checks in start and stop will handle this "real state vs cluster state" race condition. This has always been a problem for _stop as it is not a master node action and doesn’t always have the latest cluster state.

closes #45609

Relates to #45562

Backport of #45627 and #45676

[elasticmachine]

Pinging @elastic/ml-core

[benwtrent]

Don't mark this ready to merge for a little while until the initial commit has time to bake for a bit in CI.

Additionally, BWC tests should be muted in master before this is merged as calling _start in a mixed cluster will fail until master updates is bwc serialization of the request object.