elasticsearch-node fails when parsing persistent tasks

[pgomulka]

It seems that elasticsearch-node is failing when parsing ML tasks - this possibly applies to other tasks defined in plugins

Elasticsearch version 7.6

Description of the problem including expected versus actual behavior:
When running a elasticsearch-node unsafe-bootstrap or elasticsearch-node remove-customs persistent_tasks an exception is thrown.

Exception in thread "main" org.elasticsearch.common.xcontent.XContentParseException: [-1:27033] [persistent_tasks] failed to parse field [tasks]
	at org.elasticsearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:429)
	at org.elasticsearch.common.xcontent.ObjectParser.parseArray(ObjectParser.java:421)
	at org.elasticsearch.common.xcontent.ObjectParser.parseSub(ObjectParser.java:453)
	at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:284)
	at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:246)
	at org.elasticsearch.common.xcontent.ObjectParser.apply(ObjectParser.java:295)
	at org.elasticsearch.persistent.PersistentTasksCustomMetaData.fromXContent(PersistentTasksCustomMetaData.java:206)
	at org.elasticsearch.common.xcontent.NamedXContentRegistry$Entry.lambda$new$0(NamedXContentRegistry.java:63)
	at org.elasticsearch.common.xcontent.NamedXContentRegistry.parseNamedObject(NamedXContentRegistry.java:141)
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.namedObject(AbstractXContentParser.java:385)
	at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1403)
	at org.elasticsearch.gateway.PersistedClusterStateService.lambda$loadOnDiskState$1(PersistedClusterStateService.java:385)
	at org.elasticsearch.gateway.PersistedClusterStateService.consumeFromType(PersistedClusterStateService.java:442)
	at org.elasticsearch.gateway.PersistedClusterStateService.loadOnDiskState(PersistedClusterStateService.java:383)
	at org.elasticsearch.gateway.PersistedClusterStateService.loadBestOnDiskState(PersistedClusterStateService.java:331)
	at org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand.loadTermAndClusterState(ElasticsearchNodeCommand.java:101)
	at org.elasticsearch.cluster.coordination.RemoveCustomsCommand.processNodePaths(RemoveCustomsCommand.java:68)
	at org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand.processNodePaths(ElasticsearchNodeCommand.java:120)
	at org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand.execute(ElasticsearchNodeCommand.java:138)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:91)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
	at org.elasticsearch.cli.Command.main(Command.java:90)
	at org.elasticsearch.cluster.coordination.NodeToolCli.main(NodeToolCli.java:49)
Caused by: org.elasticsearch.common.xcontent.XContentParseException: [-1:27033] [tasks] failed to parse field [task]
	at org.elasticsearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:429)
	at org.elasticsearch.common.xcontent.ObjectParser.parseSub(ObjectParser.java:439)
	at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:284)
	at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:246)
	at org.elasticsearch.common.xcontent.AbstractObjectParser.lambda$declareObjectArray$7(AbstractObjectParser.java:183)
	at org.elasticsearch.common.xcontent.AbstractObjectParser.lambda$declareFieldArray$13(AbstractObjectParser.java:211)
	at org.elasticsearch.common.xcontent.AbstractObjectParser.parseArray(AbstractObjectParser.java:229)
	at org.elasticsearch.common.xcontent.AbstractObjectParser.lambda$declareFieldArray$14(AbstractObjectParser.java:211)
	at org.elasticsearch.common.xcontent.ObjectParser.lambda$declareField$9(ObjectParser.java:326)
	at org.elasticsearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:427)
	... 24 more
Caused by: org.elasticsearch.common.xcontent.XContentParseException: [-1:27033] [task] failed to parse field [xpack/ml/data_frame/analytics]
	at org.elasticsearch.common.xcontent.ObjectParser.lambda$declareNamedObjects$11(ObjectParser.java:357)
	at org.elasticsearch.common.xcontent.ObjectParser.lambda$declareNamedObjects$12(ObjectParser.java:369)
	at org.elasticsearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:427)
	... 33 more
Caused by: org.elasticsearch.common.xcontent.XContentParseException: [-1:27033] [named] failed to parse field [params]
	at org.elasticsearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:429)
	at org.elasticsearch.common.xcontent.ObjectParser.parseSub(ObjectParser.java:439)
	at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:284)
	at org.elasticsearch.persistent.PersistentTasksCustomMetaData.lambda$static$9(PersistentTasksCustomMetaData.java:97)
	at org.elasticsearch.common.xcontent.ObjectParser.lambda$declareNamedObjects$11(ObjectParser.java:355)
	... 35 more
Caused by: org.elasticsearch.common.xcontent.NamedObjectNotFoundException: unknown named object category [org.elasticsearch.persistent.PersistentTaskParams]
	at org.elasticsearch.common.xcontent.NamedXContentRegistry.parseNamedObject(NamedXContentRegistry.java:128)
	at org.elasticsearch.common.xcontent.support.AbstractXContentParser.namedObject(AbstractXContentParser.java:385)
	at org.elasticsearch.persistent.PersistentTasksCustomMetaData.lambda$static$6(PersistentTasksCustomMetaData.java:94)
	at org.elasticsearch.common.xcontent.AbstractObjectParser.lambda$declareObject$1(AbstractObjectParser.java:146)
	at org.elasticsearch.common.xcontent.ObjectParser.lambda$declareField$9(ObjectParser.java:326)
	at org.elasticsearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:427)
	... 39 more

Steps to reproduce:
To Reproduce:

1. Create and run a ml job on any data it doesn't matter.
2. When the job finishes click on Start job running in real time this will leave the job task in cluster state
3. kill -9 the node. This must be done as a graceful shutdown will close the job orderly and remove the persistent task
4. Now you have a stopped node with clusterstate containing a persistent task.
5. bin/elasticsearch-node remove-customs persistent_tasks

relates #48390

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Distributed)

[davidkyle]

@pgomulka I edited the title as the problem applies to persistent tasks in general not specifically ml tasks.

[nik9000]

Mine. I'll make sure we have a test for this so I don't break it again....

[nik9000]

Mine. I'll make sure we have a test for this so I don't break it again....

Turns out when @ywelsch pinged me about this he was just asking my opinion and he'll be fixing it.