Not All Postgrescluster Resources Deleted on Helm Uninstall

[ranchodeluxe]

Overview

Given that we do not have an OCI for the postgrescluster helm chart (as this ticket documents), I've been copying the postgres-operator-examples helm charts to be direct dependencies of my chart. Things work fine on helm install but on helm uninstall most of the postgrescluster resources are not removed (statefulsets, some deploys, some pods, configmaps, secrets)

Environment

Please provide the following details:

• Platform: GKE
• Platform Version: v1.27.13-gke.1070000
• PGO Image Tag: ubi8-16.3-3.4-0
• Postgres Version: 16
• Storage: standard-rwo pd.csi.storage.gke.io

Steps to Reproduce

REPRO

0. check out the branch related to this PR: Use PGO by Default developmentseed/eoapi-k8s#84
1. cd /helm-chart/
2. run helm install -n eoapixyz --create-namespace eoapixyz ./eoapi -f eoapi/values.yaml
3. run kubectl get pod,deploy,svc,statefulset -n eoapixyz
4. run helm uninstall eoapixyz
5. run kubectl get pod,deploy,svc,statefulset -n eoapixyz (and see there are still resources there)

EXPECTED

running kubectl get pod,deploy,svc,statefulset -n eoapixyz after running helm uninstall shouldn't show any resources in that namespace

ACTUAL

running kubectl get pod,deploy,svc,statefulset -n eoapixyz after running helm uninstall shows there are existing resources in that namespace

Logs

None

Additional Information

I have noticed interesting behaviour when just installing pgo helm chart (not as a dependency) that makes me thing other things are going on. For example, after helm install -n testing123 --create-namespace pgo ./pgo and helm uninstall pgo I cannot delete my namespace testing123 b/c the finalizer is waiting on the state of the crunchdata CRD. And I cannot delete the CRD, it just hangs. Same situation for regardless if pgo.singleNamespace is true or false

[benjaminjb]

Hello @ranchodeluxe, sorry you're running into this and I can replicate the first problem you've described with your chart. But I think this is because of how Helm charts work.

That is, my understanding of Helm chart dependencies is that they aren't installed in any particular order by chart, but rather by K8s object type. This section in the Helm docs is what I'm thinking of and provides a good example: Helm chart A with dependency B mushes all the K8s objects each chart specifies and then, from that mush, follows the usual Helm install order.

(Note: as far as I know, CRDs are always installed first and never uninstalled by Helm (docs).)

What that means, in your first case, is that the Helm chart with PGO and postgrescluster dependencies is installing those three charts (parent, pgo dependency, postgrescluster dependency) at the same time. Then, when it's uninstalling, it's uninstalling all three the same time. When I tried it, what I saw was that the PGO pod was one of the first to get uninstalled -- and without the PGO pod around, the postgrescluster cannot be cleaned up since the PGO pod had the logic for removing the finalizer.

Now, Helm charts do have a concept of hooks which can help intervene at certain points in the lifecycle of a chart, but frankly I have found them to be unhelpful in most cases. Actually, in this case, I can think of two workarounds:

Workaround 1: Two Helm charts

The problem here comes from pgo and postgrescluster Helm charts being deleted at the same time, so let's break it up:

1. remove pgo as a subchart/dependency
2. install pgo through the Helm chart available in the OCI registry first
3. install the eoapi Helm chart with the postgrescluster dependency

Then when it's time to clean up, uninstall eoapi (with the postgrescluster dependency) and then uninstall pgo.

Workaround 2: A little Kubectl

This workaround might be a little dirtier, but if you kubectl delete the postgrescluster first, then when it's gone, you can helm uninstall the parent chart and everything should be cleaned up. (This is because the kubectl delete already deleted the postgrescluster while the PGO pod was still in operation.)

[benjaminjb]

As for the second problem you describe (not being able to delete the namespace after installing the PGO Helm chart), I am not able to replicate that: I installed the Helm chart through the command you gave, and deleted it, and then I deleted the namespace just to check.

(I don't quite understand what's causing your namespace not to delete, but it can't be the CRD because CRDs aren't namespaced. So when I uninstall the helm chart and delete the testing123 namespace, when I kubectl get crd, I still see the postgresclusters.postgres-operator.crunchydata.com and other CRDs that the chart installed. Because, as noted above, Helm doesn't delete CRDs -- but also, the presence of that CRD didn't block the namespace deletion.)

I think these two comments should answer your questions, so I'm going to close this issue, but if you have other concerns about this topic, please feel free to reopen this issue and we can discuss it more.

[ranchodeluxe]

thanks @benjaminjb! Yep, all that makes sense and works in the correct ordering

[benjaminjb]

Awesome, let us know how it goes!