[ML] Datafeed does not start when allow_lazy_open is enabled

[sophiec20]

Found in 7.7.0-SNAPSHOT "build_hash" : "2f0aca992bb8c91c17603050807891cad2e41483", "build_date" : "2020-03-16T02:52:34.086738Z",

• 3 node cluster, all nodes acting as data, master and ml
• All nodes are co-located on the same 16GB VM
• "xpack.ml.max_machine_memory_percent" : 16

I have a script that creates 16 jobs in succession. Each job requires 2GB model memory.

The first 3 jobs open and the datafeeds start.
The 4th job returns opened:false and the datafeed fails to start with the following:

open job        {"opened":false}
start datafeed  {"error":{"root_cause":[{"type":"status_exception","reason":"Could not start datafeed, allocation explanation []"}],"type":"status_exception","reason":"Could not ...

In the job list, the job state is opening and the datafeed state is stopped. No errors are visible.

As one of the first 3 jobs completes, one of the opening jobs transitions its state to opened. However the datafeed remains stopped.

These are the job messages for a job that was lazy opening.

Expected behavior would be for the datafeed to be starting and for it to start once resource became available (which would happen when one of the other jobs closed, in this scenario).

Once jobs have completed, I can manually start the datafeed on one of the opened jobs and it will complete without on-screen errors. (I cannot start one of the opening jobs, which is to be expected.)

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[droberts195]

Expected behavior would be for the datafeed to be starting and for it to start once resource became available

There are two bits to this. Obviously it's a bug that the datafeed doesn't start once resource is available and that's relatively easy to fix.

Having the state being starting rather than stopped while it's waiting is a bigger problem. Although the code has a starting state, at the moment it's never used. These comments relate to that:

elasticsearch/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ml/MlTasks.java

Line 135 in 8bbbe28

// TODO: report (task != null && task.getState() == null) as STARTING in version 8, and fix side effects

elasticsearch/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/action/TransportStopDatafeedAction.java

Lines 97 to 101 in 20d4bc8

	// The STARTING state is not used anywhere at present, so this should never happen.
	// At present datafeeds that have a persistent task that hasn't yet been assigned
	// a state are reported as STOPPED (which is not great). It could be considered a
	// breaking change to introduce the STARTING state though, so let's aim to do it in
	// version 8. Also consider treating STARTING like STARTED for stop API behaviour.

So:

• The plan was to make starting an externally visible state in 8.0 to avoid screwing up any programmatic integration that doesn't expect that state in a minor version upgrade.
• We could do this sooner if we think the effects of not having it are bad enough - stopped is certainly a confusing state for a datafeed that's waiting for node assignment.
• The comment "Also consider treating STARTING like STARTED for stop API behaviour." alludes to further complications with making this change. We would need to do that if we make starting a first-class citizen, otherwise you wouldn't be able to stop a datafeed that was waiting for node assignment, that in turn would prevent you closing the job and we'd be back to the horrible deadlock situation we had 6 months ago. But the bigger problem is whether there would be any unforeseen side effects of introducing starting.

We should probably discuss offline whether introducing starting to the outside world in 7.7 would count as a breaking change or a non-breaking enhancement.

[sophiec20]

I have no desire to introduce starting if this causes excessive implementation complications. We have a tight coupling to the job/datafeed relationship, so complexity in the multi-state relationship is also something we should avoid in our future roadmap. I hadn't thought that through enough when writing the expected behaviour para above.

To clarify scope and intention of this ticket - it is to ensure that jobs can lazy open. So, let's fix the bug.

[droberts195]

I think we might need to make starting externally visible in the datafeed stats. After fixing just the lazy start side it confuses the UI if the externally visible status of the datafeed is stopped. The UI has no way to know that it should attempt to stop that datafeed.

I will make the change on Monday.