Error upgrading from (<= 6.5) to 6.7.x

[benwtrent]

While upgrading from version 6.5.x (or a previous version) to 6.7.x, there is a bug that puts a null value into the cluster metadata state in certain situations.

If there exists any ML Anomaly jobs or Datafeeds and there are NO persistent tasks in the cluster state, then the ml config migration process puts a null value into the the new cluster state and attempts to update the state. This, of course, fails as the Diff check sees the null value and throws an NPE.

This issue can be worked around in any of the following ways:

• Delete Jobs and datafeeds
• Set the cluster setting xpack.ml.enable_config_migration to false. It is a dynamic setting.
• Open any ML job so that a persistent task gets created.

before setting xpack.ml.enable_config_migration: true again, the jobs either need to be deleted, or open one of the jobs. That will then allow the migration to take place.

[elasticmachine]

Pinging @elastic/ml-core

[droberts195]

NO persistent tasks in the cluster state

... and never ever have been I think. If you have ever created a persistent task but just don’t have any right now then the corresponding custom metadata will exist, just storing the highest ever allocation ID, and then the collection that’s null when the problem occurs will instead be an empty collection.

[droberts195]

Although this cannot affect rolling upgrades to 7.x (because to do such an upgrade you have to first upgrade to 6.7), I don't see why it couldn't affect full cluster restart upgrades from 6.0-6.5 direct to 7.x. Maybe I am missing something though.

Since the fix is so tiny I think it would be best to put it into master, 7.x and 7.0 as well as 6.7. It cannot hurt even if it's unnecessary.

One more point of note: if this is theoretically a problem for full cluster restart upgrades from 6.0-6.5 direct to 7.x then at least it will not affect Cloud, because I believe Cloud is forcing all upgrades to 7.x to go via 6.7.

[benwtrent]

@droberts195 for sure. I am going to open PRs for adding the fix to 7.0, 7.X and master. I wanted to get 6.7 opened first as it was where it was discovered first.