Forgot to install a plugin can cause custom cluster state parts to be removed

[martijnvg]

If a plugin adds custom cluster state parts and the plugin isn't installed on all nodes then it could cause custom metadata to be removed by a node just by reading it.

Brought up while reviewing #21243.

[clintongormley]

On the other hand, if you uninstall a plugin, it's the only way of removing its custom metadata. Perhaps we should just recommend using the required plugins setting?

[bleskes]

my kneejerk reaction in the review was to refuse starting when you can't read a local cluster state due to the lack of a plugin and make the plugin manager to remove/archive pieces of the cluster state. This is however not trivial at all as the plugin manager need to reach into ES storage which is something we didn't do before (and may require custom knowledge in the future where current plans will make it more complicated). There is also the question if we want the plugin manager to "archive" this like we do with other config.

Perhaps we should just recommend using the required plugins setting?

yeah.. it's fairly hidden now (it took a couple of iteration to find it, knowing it exists.) Maybe we should promote it to a note in the introductions section or the management intro of the plugins.

[martijnvg]

Right, but that doesn't protected against unintended removal of custom metadata. Maybe we should go even further and require that plugins are also specified in the plugin.mandatory setting? (if a plugin isn't in the this list then we abort starting a node)

[bleskes]

We are working on addressing this as part of the node validaing in zen 2. @ywelsch @DaveCTurner do you have this captured in another place so we can close the issue?

[bleskes]

@DaveCTurner ping re ^^ ?

[DaveCTurner]

The approach we're currently investigating for Zen 2 now looks like it'll be fairly independent of this issue: it won't fix this, but nor will it significantly affect the work required to fix this.