Task Cancellation Enhancements

[Bukhtawar]

Problem

1. The task cancellation results in a ban on child tasks and marking the task cancelled in the task manager, however if the child tasks continue to run without checking on the cancellation(it does so at some points) via CancellableTask#isCancelled, the list task API continues to show the task on the task manager without an indication of whether the task/child task was cancelled, though it might eventually fail/abort.
2. The cancellation task API supports cancellation of tasks via actions/nodes. It might be good to have a bulk task cancellation API based on task ids, which might be based on task running time etc

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

I can see some value in the first idea although really IMO it's a bug not to check for cancellation frequently enough for this to matter.

For (2) I much prefer the more recent approach that lets a client cancel its own tasks by disconnecting. Permitting a user to cancel another user's tasks, or even some internal system tasks, is problematic, and letting them cancel tasks in bulk is just scaling up the problem.

[DaveCTurner]

See #51628 for info on the planned future direction of the tasks APIs, and note in particular that we have decided to implement dedicated cancellation APIs on a case-by-case basis rather than rely on the general-purpose task cancellation API.

[DaveCTurner]

A user raised with me that this first suggestion (plus #17094) would be particularly useful with force-merging. Today there's no way to cancel a force-merge on a shard but we can at least indicate in the UI that it's been cancelled while you're waiting and therefore that we won't start any further per-shard work.

[DaveCTurner]

Closing this in favour of #72907.