[ML] Add option to allow jobs to be opened when there are no ML nodes

[elasticmachine]

Original comment by @droberts195:

At present if there are no ML nodes then opening an ML job will fail immediately due to the checks in OpenJobAction.validate().

Given the way it is proposed to add lazy ML nodes to Cloud LINK REDACTED), there is a requirement to relax this check when running in Cloud.

We need to add a boolean setting that allows jobs to be opened if there are no ML nodes at all in the cluster. However, jobs should not be allowed to open when there are ML nodes in the cluster but they are all full to capacity.

The new setting will default to false, but be set to true in Cloud clusters.

A job that is opened when there are no ML nodes in the cluster will stay in the opening state until an ML node joins the cluster, at which point the job will get allocated to the ML node.

[elasticmachine]

Original comment by @droberts195:

There is another problem with ML in Cloud, and potentially both could be solved with the same solution. The other problem is as follows:

The cluster restore sequence that Cloud uses is:

1. Restore indices used by internal services (Security, Watcher, ML, Monitoring, etc.)
2. Restore global cluster state
3. Restore all other indices

The reason this has problems with ML in the mix is that step 2 restores persistent tasks and once these are restored ML will start doing stuff.

ML is currently the only GA user of persistent tasks, but I imagine rollups will have very similar problems in 6.3.

If the restore sequence could be changed to this then it would solve the problem with persistent tasks:

1. Set a transient cluster setting that tells ML not to allocate any persistent tasks, say xpack.ml.enable_persistent_task_allocations: false
2. Restore indices used by internal services (Security, Watcher, ML, Monitoring, etc.)
3. Restore global cluster state
4. Restore all other indices
5. Unset the transient cluster setting that tells ML not to allocate any persistent tasks - xpack.ml.enable_persistent_task_allocations: true

This works because transient cluster settings do not get changed by the include_global_state option in a snapshot restore.

So, once we have implemented such a setting (it doesn't exist today) then it can also use it to solve the problem this issue was originally opened for. The logic would be:

1. Cloud clusters are initially created with no ML nodes and xpack.ml.enable_persistent_task_allocations: false
2. User creates ML job
3. User opens ML job - this would normally be an error with no ML nodes, but xpack.ml.enable_persistent_task_allocations: false means it's not
4. Cloud notices that an ML job exists, adds an ML node to the cluster and, once that's ready, changes the setting of xpack.ml.enable_persistent_task_allocations to true
5. ML job gets allocated to the ML node and starts doing stuff

This approach means a single new setting can solve two different problems we currently have with Cloud deployments.

[elasticmachine]

Original comment by @droberts195:

We need to decide whether this should be a boolean setting or if it would make more sense to mimic the semantics of cluster.routing.allocation.enable as closely as possible.

Similarly for the naming; xpack.ml.allocation.enable would be closer to cluster.routing.allocation.enable than the name xpack.ml.enable_persistent_task_allocations that was suggested in the comment above.

[elasticmachine]

Original comment by @bleskes:

@droberts195 can we sync on this one? I think we should implement this on the PersistentTaskService level - something like cluster.tasks.allocations.enable: _none_ or cluster.task.allocations.enable: * . I also think we should think ahead a bit and come up with a vision to how task allocation filtering by attribute would work to more sure we don't paint ourselves into a corner when introducing this setting and have to suffer that BWC implications.

[elasticmachine]

Original comment by @droberts195:

I talked with @bleskes about implementing this via new settings for persistent tasks.

It seems that much of the functionality that gives users control over shard allocation is also relevant to persistent task allocation:

• Completely disabling persistent task allocation for the whole cluster
• Disabling persistent task allocation for nodes with certain attributes
• Disabling persistent task allocation for specific types of persistent tasks (e.g. only ML jobs, rollups, reindex, etc.)

In the 6.3 timeframe we could implement simple all or nothing:

PUT _cluster/settings
{
    "transient" : {
        "cluster.persistent_tasks.allocation.enable" : "all"
    }
}

PUT _cluster/settings
{
    "transient" : {
        "cluster.persistent_tasks.allocation.enable" : "none"
    }
}

In future this could be extended to enabling allocation of specific task names:

PUT _cluster/settings
{
    "transient" : {
        "cluster.persistent_tasks.allocation.enable" : "xpack/rollup/job"
    }
}

PUT _cluster/settings
{
    "transient" : {
        "cluster.persistent_tasks.allocation.enable" : "xpack/ml/job,xpack/ml/datafeed"
    }
}

As a further enhancement wildcarding could be added:

PUT _cluster/settings
{
    "transient" : {
        "cluster.persistent_tasks.allocation.enable" : "xpack/ml/*"
    }
}

Shard allocation by node attribute is controlled using index settings. Persistent tasks do not have equivalent per-task settings. For persistent task allocation by node attribute, it would be possible to add an optional sub-section to the existing config objects that would be made available to the persistent tasks framework via a new getSettings() method of the PersistentTaskParams interface.

For example:

PUT _xpack/ml/anomaly_detector/my_job
{
    .
    .
    .
    "allocation" : {
        "include" : "fast_cpu,medium_cpu",
        "require" : "big_memory",
        "exclude" : "something_else"
    }
}

The same could be added to rollup configuration. Then these attributes could be included in the default assignment decision (which is currently the least loaded data node). For tasks that have more complex allocation criteria, such as ML jobs, the attributes would be taken into account in addition to the other criteria. The PersistentTasksExecutor would have a method to encapsulate the allocation filtering by attribute.

The alternative would be to introduce a whole new endpoint for managing the generic settings that apply to all persistent tasks. However, although this would lead to more consistency between persistent task types, it would make it impossible to atomically set the allocation filters at the point of creating the persistent task.

[elasticmachine]

Original comment by @bleskes:

Thanks @droberts195 . This is inline with what I suggested in this ticket but I'm a bit on the fence as it is inconsistent with the cluster level settings for shard allocations. There we don't have _all or _none but have all and none (among other things). The main reason is that we never really needed to fully prevent allocation of a specific index and we also have more values there. Concretely:

PUT _cluster/settings {
	"persistent": {
	  "cluster.tasks.allocation.enable": "none/all"
	}
}`

I tend to say we should go with consistency (i.e., none and not _none). We managed to get so far without people need to disable allocation on an index level. I wonder if we'll ever need it for persistent tasks.

[elasticmachine]

Original comment by @droberts195:

@bleskes it sounds like all and none without underscores are better for consistency with indices. I edited my comment above to reflect this to make it easier for others to review.

@imotov @martijnvg @polyfractal @jasontedor do you have any comments on the proposal in LINK REDACTED

[elasticmachine]

Original comment by @bleskes:

/cc @tlrx

[elasticmachine]

Original comment by @martijnvg:

@droberts195 The proposal as is now looks good to me. Like @bleskes I don't think we will need to more fine grained control in the future.

[elasticmachine]

Original comment by @polyfractal:

++ to the short-term all/none plan as outlined.

I'm on the fence about the finer grained control. Could be useful, or maybe not used at all. Not sure. No strong opinion from me :)

And ++ to any future allocation control going on the configuration object of each individual persistent task. We'll have to keep them consistent manually, but I think that's preferable to the complexity (both code- and user-complexity) of management by some unified endpoint. Makes more sense to configure those details when you configure the Datafeed/Rollup/whatever, instead of after creation with a central endpoint

[elasticmachine]

Original comment by @bleskes:

And ++ to any future allocation control going on the configuration object of each individual persistent task. We'll have to keep them consistent manually, but I think that's preferable to the complexity (both code- and user-complexity) of management by some unified endpoint. Makes more sense to configure those details when you configure the Datafeed/Rollup/whatever, instead of after creation with a central endpoint

I guess we'll have to see how it goes. I guess at the first step we'll see to control all jobs of the same type. Later on we may need finer per job control. The later was already described. For the former I mentioned a wildcard approach (colliding with all/none and hence the need for _all) but it can be easy as well if we stopped using / in job names to namespace but rather .. For example:

PUT _cluster/settings {
	"persistent": {
	  "cluster.persistent_tasks.xpack.ml.allocation.enable": "none/all"
	}
}`

and replacing xpack.ml with any other task type.

[elasticmachine]

Original comment by @polyfractal:

Sorry, I wasn't clear in my comment. In that last portion, I was referring to where jobs are allocated, similar to shard allocation filtering. E.g. in @droberts195 proposal it was this syntax:

    "allocation" : {
        "include" : "fast_cpu,medium_cpu",
        "require" : "big_memory",
        "exclude" : "something_else"
    }

Which I think makes sense to put on the individual job configuration objects (if or when we decide to add this sort of functionality), rather than some kind of global endpoint to control job location after the job is configured.

I think the control of when jobs are allocated is fine to leave at enable: all/none and done across all jobs (as currently proposed), no need to control individual jobs imo.

Sorry for the confusion :)

[elasticmachine]

Original comment by @bleskes:

hehe, I think I wasn't clear too in my example. Part of what I mean is that I think we'll first see a need to say "allocated all ml jobs to this nodes, all reindex to those". We may want to do it per job but also maybe per type. This would look like:

PUT _cluster/settings {
	"persistent": {
	  "cluster.persistent_tasks.xpack.ml.allocation.include.box": "big"
	}
}`

Anyway, time will tell. I feel that the direction we're heading is good and will give us what we need.

[elasticmachine]

Original comment by @suyograo:

@droberts195 is there an issue for enabling/disabling the persistent tasks -- I'd like to track it from the Cloud side. Or are we using this issue for adding the enhancement?

[elasticmachine]

Original comment by @suyograo:

@droberts195 @sophiec20 also, will this be available in 6.3?

[elasticmachine]

Original comment by @bleskes:

@suyograo here's the relevant PR on the ES side: #29137 . It will be part of 6.3