[ML] Add the ability to delete a job asynchronously

[droberts195]

Deleting a job can take a long time if it has many results and is using a shared index.

The operations involved in deleting a job are already grouped into a task. Usually APIs that kick off tasks offer the option to wait_for_completion or not. However, the delete job API does not; the endpoint always waits for completion.

As part of the "Optimizing job deletion" project for 6.5 we should add a wait_for_completion flag to the delete job endpoint (default true to match historic behaviour). If set to false the endpoint should return as soon as the deletion task is started, and return the task ID in its response.

[droberts195]

#29769 could also be considered while changes are being made for this issue.

[droberts195]

Some other things to consider that we discussed verbally:

1. Can the deletion task be made a persistent task? One benefit is that if the node that's deleting the job is killed part way through the deletion then the job won't be left half deleted. The other benefit is that when job configs have moved to indices we'll have something in cluster state (namely the deletion persistent task) that we can check for to find out if the job is in the process of being deleted.
2. Should we queue up deletes so that we don't have lots running in parallel against the same index? Maybe the answer should be yes if wait_for_completion=false and no if wait_for_completion=true?

[dimitris-athanasiou]

The two points above have been discussed. I post remarks here:

1. Using a persistent task is too big a hammer for this. It brings with it a lot of complexity for dealing with an issue that seems to be an edge case at the moment. Also, delete job is internally using delete-by-query for which there are considerations to turn into a persistent task itself. The result of all the above is we should not proceed with making delete-job a persistent task.
2. It might be a good idea to wait and measure the performance issue after our current changes have baked in (deletion of index for dedicated indices, async task which prevents force-deletes from doing any work even when the first request failed). If we do find we need to further optimize, we should also consider reducing the slices of the delete-by-query.