`GET _cat/pending_tasks` will return some pending tasks that have been executed

[kkewwei]

ES_VERSION: 7.3.1
Description of the problem including expected versus actual behavior:
We can get pending tasks from cluster by GET _cat/pending_tasks, it should return the tasks that status are pending or starting. Actually, it will return some pending tasks that have being executed.

As we know that master will receive the requests from data nodes that shards have been started, then it will update ClusterState about those shards. If there are some another shard-started tasks pending at the same time, master will batch execute those shard-started tasks together, the processed status of shard-started tasks will been set true, but they are still exist in PrioritizedEsThreadPoolExecutor.workQueue, so if we get the pending tasks by GET _cat/pending_tasks, it will return the shard-started tasks that have been executed.

Solution:
If we should filter those shard-started tasks whether they have been executed before return?

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Cluster Coordination)

[DaveCTurner]

It's not clear to me what the actual problem is here. The tasks do indeed remain in the queue after they have been processed if that happened as part of a batch, so it seems reasonable to me to report them in the output of the pending tasks API. Note that they're only reported until they are themselves processed, which I expect to happen fairly soon afterwards. @kkewwei can you explain in more detail why you need processed tasks to be removed from this list more quickly?

Note also that shard-started tasks (specifically) are processed more quickly in 7.4.0 thanks to #44433.

[kkewwei]

During cluster operation, there will be lots of shard-started tasks, those shard-started tasks that have been processed are still exist in pending list, @DaveCTurner it's just a little confusing. we should reflect the real situation of the cluster, to achieve the goal, we just need to filter it before displaying. Of cause I'm not sure whether it is a bug.

[DaveCTurner]

The pending tasks API exposes some fairly deep internals of the system and I don't think we should be filtering it as you describe. Indeed, I would find it more confusing to filter it like this. There are other ways to get no-op shard-started tasks in the list, for instance here:

elasticsearch/server/src/main/java/org/elasticsearch/indices/cluster/IndicesClusterStateService.java

Lines 646 to 648 in 4749ba6

	shardStateAction.shardStarted(shardRouting, primaryTerm, "master " + nodes.getMasterNode() +
	" marked shard as initializing, but shard state is [" + state + "], mark shard as started",
	SHARD_STATE_ACTION_LISTENER, clusterState);

I think you will need to adjust your usage of this API to account for such things, so I'm closing this.