Optimize cluster mutations by reusing PVCs when replacing nodes

[sebgl]

Reuse volumes on topology changes.

[pebrc]

@barkbay I think you are working on this? At least it seems closely related to #311

[pebrc]

This applies to such changes only that require a new pod. Everything else should be handled via in place mutation (keeping the pod around, changing configuration, restarting ES)

For those cases where we need to recreate (e.g. major version upgrade) we should adopt the following strategy:

• try reuse if replicas available elsewhere, otherwise grow and shrink
• if reuse fails, fall back to grow and shrink with migration
• if reuse succeeds but Elasticsearch does fail after the upgrade?
  • we surface the condition e.g. via Elasticsearch health or pod status
  • manual intervention via kubectl exec or similar

as dicussed with @\bleskes

[pebrc]

We discussed today breaking the implementation of this down into separate phases:

• start with pod reuse reviving WIP - TLS to non-TLS license mutations with full cluster restart #604
• implement volume reuse (with minimal safety checks but with a strategy attribute in the spec to turn volume reuse off)
• improve on the initial implementation with additional safety checks (e.g. fall back to grow and shrink when pod cannot be scheduled)