keepTLSSecret value not being honoured

[lcaproni-pp]

Describe the bug

There was an update here #2264 - to allow users of the helm chart to keep the TLS secrets as they are and to stop non-empty diffs running every plan/apply. However upon setting the value the chart still generates diffs for the TLS certs.

Steps to reproduce

1 - Install helm chart & set the keepTLSSecret value to true
2 - Run a helm diff after installation
3 - Observe output and notice that diffs are still being generated

Expected outcome
The webhook secret that already contains the TLS certs should be re-used when using the keepTLSSecret value within the helm chart.

It could be that i'm missing some configuration or something but there is nothing that seems obvious to me.

Environment

• AWS Load Balancer controller version - v2.3.0 -Helm chart v1.3.1
• Kubernetes version - 1.21
• Using EKS (yes/no), if so version? Yes - 1.21

[oliviassss]

@lcaproni-pp Can you try to use helm upgrade after setting the keepTLSSecret value to true? And then check if the key and cert in aws-load-balancer-tls are unchanged.

[oliviassss]

@lcaproni-pp we use lookup function to query the current secret, however, according to the doc:

Helm is not supposed to contact the Kubernetes API Server during a helm template or a helm install|update|delete|rollback --dry-run, so the lookup function will return an empty list (i.e. dict) in such a case.

And helm diff is actually using helm upgrade --debug --dry-run (ref link), that's why lookup doesn't work in helm diff either, so the TLS secret changed in the output.
Please try to use helm upgrade directly as a workaround, and let us know if you have further questions.

[mglotov]

We use https://github.com/hashicorp/terraform-provider-helm to install charts into clusters and face the same issue.
This is the output of terraform apply command:

  ~ resource "helm_release" "aws_loadbalancer_controller" {
        id                         = "aws-load-balancer-controller"
      ~ manifest                   = jsonencode(
          ~ {
              ~ ing/secret/v1/aws-load-balancer-tls                                                                                             = {
                  ~ data       = {
                      ~ ca.crt  = "KHNlbnNpdGl2ZSB2YWx1ZSBiNjg3ZTg4OTM5ZDQwYWFjKQ==" -> "KHNlbnNpdGl2ZSB2YWx1ZSAzZjA3M2I1MDY3MGJiZTkxKQ=="
                      ~ tls.crt = "KHNlbnNpdGl2ZSB2YWx1ZSBlODUzODMyOWQ5OTkwZTA0KQ==" -> "KHNlbnNpdGl2ZSB2YWx1ZSBiMDcyY2E5NDYzN2NmMDkzKQ=="
                      ~ tls.key = "KHNlbnNpdGl2ZSB2YWx1ZSA0MWQ4MjFkZDAyMjllYjg0KQ==" -> "KHNlbnNpdGl2ZSB2YWx1ZSBiMzA4NDFmMWZlNjYzMThiKQ=="
                    }
                    # (4 unchanged elements hidden)
                }

Each time when we run terraform apply certs are re-generated again.

Environment
terraform helm provider version: 2.3.0 (https://registry.terraform.io/providers/hashicorp/helm/latest/docs#experiments is enabled)
AWS Load Balancer controller version - v2.3.0 -Helm chart v1.3.1
Kubernetes version - 1.21
Using EKS (yes/no), if so version? Yes - 1.21
keepTLSSecret is set to true

[lcaproni-pp]

@oliviassss Thanks for this, I tried doing

helm upgrade --debug --dry-run aws-load-balancer-controller eks/aws-load-balancer-controller -f values.yaml

I had to manually compare the certs with the ones that are already stored on the cluster and they didn't match up at least from what I seen so my deduction was that they would have been changing from what was already stored on the cluster. I will say also that our pipelines are doing a helm install/apply || helmfile apply when the PR is merged and checking the pipeline logs the chart was trying to update the values.

Values file contents just for reference:

clusterName: CLUSTERNAME

autoDiscoverAwsVpcID: true

awsRegion: eu-west-1

keepTLSSecret: true

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: XXXXXXXX

[ypicard]

Can confirm this does not work when running helmfile apply.

[tamirhad]

not working with Pulumi helm module either

[kishorj]

With the recent chart version v1.3.x, we've also added support for directly specifying the TLS certificates/keys during chart install. Here is the snippet from the values.yaml:

# webhookTLS specifies TLS cert/key for the webhook
webhookTLS:
  caCert:
  cert:
  key:

This method doesn't depend on querying the cluster for existing secret, and provide consistent results with the --dry-run or helm template without --validate. Please see if this option is useful in this situation. Note: you need to specify all three attributes - caCert, cert and the key.

[ypicard]

Is this the required way to do things, or is it actually bug to still view a diff when keepTLSSecret is set to true?

[jdomag]

keepTLSSecret option doesn't either work in 2.3 release with newest helm chart - we use argocd for deployments.

[lcaproni-pp]

Siding with @ypicard on this one here, if this is the recommended way to go then the keeptls feature should be removed and docs should be updated so users know to do this. I know it's a feature but personally I didn't want to go through the hassle of doing this X times and then every X time I deploy this helm chart into a new cluster. If that's how it has to be then fair enough but since it wasn't previously I hoped it would stay the same.

[tyrken]

@jdomag I haven't tried it yet (I use helmfile not ArgoCD for deploying this project, nor have tried the 2.3 chart myself yet) but there supposedly is an ArgoCD workaround. See this comment to let ArgoCD ignore changes. Please let us know if this still works or the config needs updating for the latest helm chart.

Feedback on 2.3 - using lookup in the helm templating will not work in many non-vanilla helm scenarios & is probably the cause for the continuing problems (this issue) - please reimplement the fix.

[jdomag]

@tyrken
Yep I saw that, however adding ignore to ArgoCD is just a workaround, not a problem solution. Thanks for a hint though!

[lcaproni-pp]

Hey @oliviassss @kishorj is there any update on this at all? Is the offical recommended way to do this generate our own secrets and store them inside the cluster or is there a bug that needs addressing? Happy to help out in some way if possible :)