aws-load-balancer-webhook-service error for non alb Ingresses

[gazal-k]

We are trying to migrate from ingress-nginx to aws-load-balancer-controller. We are starting by just installing the controller chart. The plan is to template our applications to use the new ingress.class alb and then migrate them.

But after installing aws-load-balancer-controller, we are seeing errors on our existing applications like:

cannot patch "app1-ingress" with kind Ingress: Internal error occurred: failed calling webhook "vingress.elbv2.k8s.aws": Post https://aws-load-balancer-webhook-service.kube-system.svc:443/validate-networking-v1beta1-ingress?timeout=10s: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "aws-load-balancer-controller-ca"): cannot patch "app1-ingress" with kind Ingress: Internal error occurred: failed calling webhook "vingress.elbv2.k8s.aws": Post https://aws-load-balancer-webhook-service.kube-system.svc:443/validate-networking-v1beta1-ingress?timeout=10s: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "aws-load-balancer-controller-ca")

app1-ingress still uses kubernetes.io/ingress.class: nginx. Can we skip the webhook from modifying those?

[M00nF1sh]

@gazal-k
It seems the webhook's CA is configured incorrectly. If you install from helm, did you reverted the webhookConfiguration/cert Secret to make them drift?

The webhook for Ingress is merely validating changes(for security features we included with IngressClassParams support), and it'll just permit any changes for other Ingresses using other IngressClasses.

It's fine to delete the ValidatingWebhookConfiguration if it's a single cluster use, but you should fix the certificate ideally.

[gazal-k]

I'm not entirely sure what may have happened. Because this wasn't me personally who installed the chart. @anilkumarpasupuleti do u know if we reverted webhookConfiguration/cert Secret during installation?

I just checked the installation instructions now. It's not a straight helm install, is it? Do the CRDs have to be done manually first? Can not doing that cause this issue?

[gazal-k]

Is this the Secret we are talking about: https://github.com/aws/eks-charts/blob/master/stable/aws-load-balancer-controller/templates/webhook.yaml#L131-L159 ?

[gazal-k]

We also checked this somewhat related #1909. Verified that the caBundle in the webhooks matched the Secret; aws-load-balancer-tls

[gazal-k]

It's fine to delete the ValidatingWebhookConfiguration if it's a single cluster use, but you should fix the certificate ideally.

Is that validatingwebhookconfigurations/aws-load-balancer-webhook? I'm no sure what you mean by "single cluster use". We plan on installing aws-load-balancer-controller on each of our clusters. Wasn't aware a single installation can manage across clusters. (not that we'd need it even so)

[anilkumarpasupuleti]

This issue does not exist on chart version 1.1.6, its on k8s v1.18

[kishorj]

@anilkumarpasupuleti, the validating webhook is included in chart v1.20 (app v2.2.0) and later.

[gazal-k]

We got the error in chart version 1.20. We tried downgrading and obviously, no validating webhook no errors.

Anil was just mentioning that we're trying all of this in EKS 1.18. (not to be confused with the chart version)

[kishorj]

@gazal-k
could you compare the caBundle configured in your validating webhook aws-load-balancer-webhook and the secret aws-load-balancer-tls?

caBundle configured in webhook

kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io aws-load-balancer-webhook -ojsonpath={.webhooks[0].clientConfig.caBundle}  | base64 -d  | openssl x509 -noout -text

CA certificate from the secret

kubectl get secret -n kube-system  aws-load-balancer-tls -ojsonpath="{.data.ca\.crt}" |base64 -d   |openssl x509 -noout -text

Also verify the webhook certificate is issued by the configured CA

get secret -n kube-system  aws-load-balancer-tls -ojsonpath="{.data.tls\.crt}" |base64 -d   |openssl x509 -noout -text

[gazal-k]

• The caBundle in the webhook & CA certificate in the secret match.
• webhook certificate is issued by configured CA.

I was able to verify installing & updating apps using other ingress controllers using helm & helmfile just fine. It seems like a hemlfile execution from our CI/CD pipeline is where we are seeing the error. This might be a tooling issue. We'll try updating our pipeline containers with newer versions of helm & helmfile & see if that helps.

Thanks @kishorj & @M00nF1sh

[darrenwhighamfd]

I have also had this issue when using Polaris https://artifacthub.io/packages/helm/fairwinds-stable/polaris on version 4.0.4

Error: UPGRADE FAILED: cannot patch "polaris" with kind Ingress: Internal error occurred: failed calling webhook "vingress.elbv2.k8s.aws": Post "https://aws-load-balancer-webhook-service.kube-system.svc:443/validate-networking-v1beta1-ingress?timeout=10s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "aws-load-balancer-controller-ca")

We use cert-manager as detailed with the Polaris documentation and currently running EKS 1.20

[M00nF1sh]

@darrenwhighamfd Is this issue still happening in your cluster? have you checked whether the cert stored in secret matches the CA configured in aws-load-balancer-webhook ValidatingWebhookConfiguration?

[DmitriyStoyanov]

Faced with the same issue, with 2 ingress controllers installed on one cluster aws-load-balancer controller and nginx-ingress controller with
We have automated CI with updating every week for new versions from helm, and after some upgrade during creation of ingresses with annotation kubernetes.io/ingress.class: "nginx" we faced with issue:
aws-load-balancers validation webhook started to handle such requests, with error, related to certificate.

Error: UPGRADE FAILED: cannot patch "kube-prometheus-stack-grafana" with kind Ingress: Internal error occurred: failed calling webhook "vingress.elbv2.k8s.aws": Post "https://aws-load-balancer-webhook-service.kube-system.svc:443/validate-networking-v1beta1-ingress?timeout=10s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "aws-load-balancer-controller-ca")

Fixed the issue by uninstallation of helm chart aws-load-balancer controller and installing it back again.

Hm, just now I see, that issue spontaneously still exist during deploy of some helm charts with Ingress. Triggering upgrade again, deploy successfully. So issue still exist.
Looks like web hook validation do not check annotations such as kubernetes.io/ingress.class or ingressClass in Ingress spec

[darrenwhighamfd]

@M00nF1sh yes we still have this issue, currently we have had to revert to 1.1.6
Similar to @DmitriyStoyanov this issue did not every time happen but spontaneously. One out of every three or so runs. The cert and secret look to match.

[gazal-k]

I guess that's our experience too. Maybe it wasn't the tooling. It was just that the issue was sporadic. We had verified the certs and secrets matching; #2071 (comment)