Question about E2E Encryption with NGF

[EmekaNnaji]

How can I enable E2E encryption in AWS EKS using NGF? I've tried using a TLS listener with a cert with a TLS TG to the Nginx Pod. Basically, I want Client (TLS) -> NLB (TLS) -> Nginx (Terminate TLS) -> Backend.I keep seeing client closed connection while SSL handshaking, client: x.x.x.x, server: 0.0.0.0:443. I'm able to get my flow working my editing the generated Nginx Conf, specifically this block from:

server {
    listen 443 ssl default_server;
    listen [::]:443 ssl default_server;
    ssl_reject_handshake on;
}

to

server {
    listen 443 ssl default_server;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_nginx-gateway_bootstrap-ingress-gateway-cert.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_nginx-gateway_bootstrap-ingress-gateway-cert.pem;
    listen [::]:443 ssl default_server;
    return 444;
}

Then disabling SNI:

apiVersion: gateway.nginx.org/v1alpha2
kind: NginxProxy
metadata:
  name: disable-sni-proxy
  namespace: nginx-gateway
spec:
  disableSNIHostValidation: true

I feel like there's definitely a better way to accomplish this, just not sure how.

[bjee19]

Hey @EmekaNnaji, thanks for opening this discussion.

I was doing some digging around and I think that if you follow the HTTPS termination guide, and set up your NLB to be TCP, that should enable e2e encryption.

To avoid the TLS being terminated on the NLB, having it be TCP makes it so the NLB doesn't touch anything related to the TLS, however the TLS handshake should continue from Client(tls) -> NLB(TCP just forwards request, data should stay encrypted) -> NGINX(Terminate TLS) -> backend.

I'm think this solution still enables e2e encryption, basically off loading the tls configuration onto NGF and the gateway api resources instead of on the NLB.

Please try it out and let me know if it works, or if you have any other questions.

Here's a stackoverflow link that is related: https://stackoverflow.com/questions/67210192/aws-network-load-balancer-ssl-passthrough.

[EmekaNnaji]

Hey, Thanks for the quick reply. For reason's outside my control, I'm stuck with using self signed certs currently. Since those would be untrusted by the browser, I'm trying to circumvent that by slapping an ACM cert on the NLB.

[bjee19]

@EmekaNnaji, ok I've looked into it a bit more and from some responses from GPT and this similar stackoverflow link https://stackoverflow.com/questions/70677589/how-to-use-aws-nlb-with-nginx-ingress-controller-for-ssl. I think you can keep the ACM cert on the NLB, though this will result in a tls termination on the NLB. The NLB will then have to open a second tls session to the NGINX Service and I think you can do this by putting the following annotations on the NGINX Service.

annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance" # or ip
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "v2"

    # Turn the listener into TLS and attach the ACM cert
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:...:certificate/XXXXXXXX
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"

    # Tell the controller that the **target group** must also be TLS
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tls"

Or something similar to the above.

You can provide these annotations to the NGINX Service by modifying the gateway.spec.infrastructure fields on the Gateway -- see guide here.

This is starting to move outside of my knowledge, but I hope some of this is looking familiar to you and you can build a solution similar to what I provided.

[EmekaNnaji]

@bjee19 - Thanks for the pointers. I should note I've done that part.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: cafe-gateway
  namespace: nginx-gateway 
spec:
  gatewayClassName: nginx
  infrastructure:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "ssl"    
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:xxxxxxxx
  listeners:
    - name: https
      port: 443
      protocol: HTTPS
      hostname: '*.example.com'
      allowedRoutes:
        namespaces:
          from: All
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: bootstrap-ingress-gateway-cert
            namespace: nginx-gateway

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: argocdv2
  namespace: nginx-gateway
spec:
  hostnames:
  - argocd.example.com
  parentRefs:
  - name: cafe-gateway
    namespace: nginx-gateway
    sectionName: https
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: argocd-server
      port: 80
      namespace: argocd

The problem is specifically comms between the NLB and Nginx. Terminating SSL on the NLB then re-encrypting and passing to the Nginx pod is where the breakdown is. SNI gets dropped at the NLB (My assumption), so all traffic from the NLB to the Nginx Pod match the default server block that then drops the traffic because of

server {
    listen 443 ssl default_server;
    listen [::]:443 ssl default_server;
    ssl_reject_handshake on;
}

I wondering if there's a sane way to make updates to the nginx config besides my current solution of going into the nginx pod and manually updating that problematic block from what it is above to:

server {
    listen 443 ssl default_server;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_nginx-gateway_bootstrap-ingress-gateway-cert.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_nginx-gateway_bootstrap-ingress-gateway-cert.pem;
    listen [::]:443 ssl default_server;
    return 444;
}

[bjee19]

@EmekaNnaji, ok I'm starting to understand more. Just verifying, what version of NGF are you running? If it is 2.X+, you shouldn't be able to modify the nginx conf like such, so I'm wondering how you're doing that.

We have a better way of injecting nginx conf through the use of SnippetsFilters, but that doesn't have the ability to remove nginx conf (or rewrite that default_server block).

Could you also paste the full nginx conf that makes your traffic flow?

[EmekaNnaji]

@bjee19 - I'm running version 2.1.2:
Here's the generated nginx conf that doesn't work:

# configuration file /etc/nginx/nginx.conf:
load_module /usr/lib/nginx/modules/ngx_http_js_module.so;
include /etc/nginx/main-includes/*.conf;

worker_processes auto;

pid /var/run/nginx/nginx.pid;

events {
  include /etc/nginx/events-includes/*.conf;
}

http {
  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/mime.types;
  js_import /usr/lib/nginx/modules/njs/httpmatches.js;

  default_type application/octet-stream;

  proxy_headers_hash_bucket_size 512;
  proxy_headers_hash_max_size 1024;
  server_names_hash_bucket_size 256;
  server_names_hash_max_size 1024;
  variables_hash_bucket_size 512;
  variables_hash_max_size 1024;

  sendfile on;
  tcp_nopush on;

  server_tokens off;

  server {
    listen unix:/var/run/nginx/nginx-status.sock;
    access_log off;

    location /stub_status {
        stub_status;
    }
  }
}

stream {
  variables_hash_bucket_size 512;
  variables_hash_max_size 1024;

  map_hash_max_size 2048;
  map_hash_bucket_size 256;

  log_format stream-main '$remote_addr [$time_local] '
                         '$protocol $status $bytes_sent $bytes_received '
                         '$session_time "$ssl_preread_server_name"';
  access_log /dev/stdout stream-main;
  include /etc/nginx/stream-conf.d/*.conf;
}

# configuration file /etc/nginx/main-includes/main.conf:

error_log stderr info;



# configuration file /etc/nginx/events-includes/events.conf:

worker_connections 1024;

# configuration file /etc/nginx/conf.d/http.conf:
http2 on;

# Set $gw_api_compliant_host variable to the value of $http_host unless $http_host is empty, then set it to the value
# of $host. We prefer $http_host because it contains the original value of the host header, which is required by the
# Gateway API. However, in an HTTP/1.0 request, it's possible that $http_host can be empty. In this case, we will use
# the value of $host. See http://nginx.org/en/docs/http/ngx_http_core_module.html#var_host.
map $http_host $gw_api_compliant_host {
    '' $host;
    default $http_host;
}

# Set $connection_header variable to upgrade when the $http_upgrade header is set, otherwise, set it to close. This
# allows support for websocket connections. See https://nginx.org/en/docs/http/websocket.html.
map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

## Returns just the path from the original request URI.
map $request_uri $request_uri_path {
  "~^(?P<path>[^?]*)(\?.*)?$"  $path;
}

# NGINX health check server block.
server {
    listen 8081;
    listen [::]:8081;

    location = /readyz {
        access_log off;
        return 200;
    }
}


js_preload_object matches from /etc/nginx/conf.d/matches.json;
server {
    listen 443 ssl default_server;
    listen [::]:443 ssl default_server;
    ssl_reject_handshake on;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_nginx-gateway_bootstrap-ingress-gateway-cert.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_nginx-gateway_bootstrap-ingress-gateway-cert.pem;

    server_name argocd.example.com;


    location / {






        proxy_http_version 1.1;
        proxy_set_header Host "$gw_api_compliant_host";
        proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for";
        proxy_set_header X-Real-IP "$remote_addr";
        proxy_set_header X-Forwarded-Proto "$scheme";
        proxy_set_header X-Forwarded-Host "$host";
        proxy_set_header X-Forwarded-Port "$server_port";
        proxy_set_header Upgrade "$http_upgrade";
        proxy_set_header Connection "$connection_upgrade";
        proxy_pass http://argocd_argocd-server_80$request_uri;



    }
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_nginx-gateway_bootstrap-ingress-gateway-cert.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_nginx-gateway_bootstrap-ingress-gateway-cert.pem;

    server_name ~^;


    location / {



        return 404 "";



        proxy_http_version 1.1;
    }
}

server {
    listen unix:/var/run/nginx/nginx-503-server.sock;
    access_log off;

    return 503;
}

server {
    listen unix:/var/run/nginx/nginx-500-server.sock;
    access_log off;

    return 500;
}


upstream argocd_argocd-server_80 {
    random two least_conn;
    zone argocd_argocd-server_80 512k;


    server 10.6.67.136:8080;
    server 10.6.51.14:8080;




}

upstream invalid-backend-ref {
    random two least_conn;


    server unix:/var/run/nginx/nginx-500-server.sock;




}





# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/avif                                       avif;
    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/wasm                                 wasm;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /etc/nginx/stream-conf.d/stream.conf:


server {
    listen unix:/var/run/nginx/connection-closed-server.sock;
    return "";
}

here's the modified conf that works:

# WORKING NGINX GATEWAY FABRIC CONFIG - DOUBLE TLS SETUP
# Issue Fixed: Removed ssl_reject_handshake from default server block and added SSL cert
# This allows NLB -> TLS -> NGINX Gateway -> TLS termination -> HTTP -> ArgoCD

# configuration file /etc/nginx/nginx.conf:
load_module /usr/lib/nginx/modules/ngx_http_js_module.so;
include /etc/nginx/main-includes/*.conf;

worker_processes auto;

pid /var/run/nginx/nginx.pid;

events {
  include /etc/nginx/events-includes/*.conf;
}

http {
  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/mime.types;
  js_import /usr/lib/nginx/modules/njs/httpmatches.js;

  default_type application/octet-stream;

  proxy_headers_hash_bucket_size 512;
  proxy_headers_hash_max_size 1024;
  server_names_hash_bucket_size 256;
  server_names_hash_max_size 1024;
  variables_hash_bucket_size 512;
  variables_hash_max_size 1024;

  sendfile on;
  tcp_nopush on;

  server_tokens off;

  server {
    listen unix:/var/run/nginx/nginx-status.sock;
    access_log off;

    location /stub_status {
        stub_status;
    }
  }
}

stream {
  variables_hash_bucket_size 512;
  variables_hash_max_size 1024;

  map_hash_max_size 2048;
  map_hash_bucket_size 256;

  log_format stream-main '$remote_addr [$time_local] '
                         '$protocol $status $bytes_sent $bytes_received '
                         '$session_time "$ssl_preread_server_name"';
  access_log /dev/stdout stream-main;
  include /etc/nginx/stream-conf.d/*.conf;
}

# configuration file /etc/nginx/main-includes/main.conf:

error_log stderr info;



# configuration file /etc/nginx/events-includes/events.conf:

worker_connections 1024;

# configuration file /etc/nginx/conf.d/http.conf:
http2 on;

# Set $gw_api_compliant_host variable to the value of $http_host unless $http_host is empty, then set it to the value
# of $host. We prefer $http_host because it contains the original value of the host header, which is required by the
# Gateway API. However, in an HTTP/1.0 request, it's possible that $http_host can be empty. In this case, we will use
# the value of $host. See http://nginx.org/en/docs/http/ngx_http_core_module.html#var_host.
map $http_host $gw_api_compliant_host {
    '' $host;
    default $http_host;
}

# Set $connection_header variable to upgrade when the $http_upgrade header is set, otherwise, set it to close. This
# allows support for websocket connections. See https://nginx.org/en/docs/http/websocket.html.
map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

## Returns just the path from the original request URI.
map $request_uri $request_uri_path {
  "~^(?P<path>[^?]*)(\?.*)?$"  $path;
}

# NGINX health check server block.
server {
    listen 8081;
    listen [::]:8081;

    location = /readyz {
        access_log off;
        return 200;
    }
}


js_preload_object matches from /etc/nginx/conf.d/matches.json;

# FIXED: Default server block now has SSL cert and returns 444 instead of ssl_reject_handshake
server {
    listen 443 ssl default_server;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_nginx-gateway_bootstrap-ingress-gateway-cert.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_nginx-gateway_bootstrap-ingress-gateway-cert.pem;
    listen [::]:443 ssl default_server;
    return 444;
}

# Main ArgoCD server block
server {
    listen 443 ssl;
    listen [::]:443 ssl;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_nginx-gateway_bootstrap-ingress-gateway-cert.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_nginx-gateway_bootstrap-ingress-gateway-cert.pem;

    server_name argocd-emeka.dragonflyft.com;


    location / {
        proxy_http_version 1.1;
        proxy_set_header Host "$gw_api_compliant_host";
        proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for";
        proxy_set_header X-Real-IP "$remote_addr";
        proxy_set_header X-Forwarded-Proto "$scheme";
        proxy_set_header X-Forwarded-Host "$host";
        proxy_set_header X-Forwarded-Port "$server_port";
        proxy_set_header Upgrade "$http_upgrade";
        proxy_set_header Connection "$connection_upgrade";
        proxy_pass http://argocd_argocd-server_80$request_uri;
    }
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_nginx-gateway_bootstrap-ingress-gateway-cert.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_nginx-gateway_bootstrap-ingress-gateway-cert.pem;

    server_name ~^;


    location / {
        return 404 "";
        proxy_http_version 1.1;
    }
}

server {
    listen unix:/var/run/nginx/nginx-503-server.sock;
    access_log off;

    return 503;
}

server {
    listen unix:/var/run/nginx/nginx-500-server.sock;
    access_log off;

    return 500;
}

# Backend upstream - connects directly to ArgoCD pods
upstream argocd_argocd-server_80 {
    random two least_conn;
    zone argocd_argocd-server_80 512k;

    server 10.6.67.136:8080;
    server 10.6.51.14:8080;
}

upstream invalid-backend-ref {
    random two least_conn;

    server unix:/var/run/nginx/nginx-500-server.sock;
}

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/avif                                       avif;
    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/wasm                                 wasm;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /etc/nginx/stream-conf.d/stream.conf:

server {
    listen unix:/var/run/nginx/connection-closed-server.sock;
    return "";
}

Main difference between the 2 being what I pointed out prior.

Here's what I'm seeing in the logs:
2025/09/30 00:12:04 [info] 805#805: *27244 client closed connection while SSL handshaking, client: x.x.x.x, server: 0.0.0.0:443

[bjee19]

Hey @EmekaNnaji, thanks for posting the full nginx conf, this helps a lot.

I'm wondering in the fixed version, how traffic actually reaches the backend. From the information I gathered online, when the nlb terminates ssl from the client and re-encrypts it to nginx the sni should be getting dropped (what you were already thinking). So when the request reaches nginx it should never reach the argocd block you posted:

# Main ArgoCD server block
server {
    listen 443 ssl;
    listen [::]:443 ssl;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_nginx-gateway_bootstrap-ingress-gateway-cert.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_nginx-gateway_bootstrap-ingress-gateway-cert.pem;

    server_name argocd-emeka.dragonflyft.com;


    location / {
        proxy_http_version 1.1;
        proxy_set_header Host "$gw_api_compliant_host";
        proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for";
        proxy_set_header X-Real-IP "$remote_addr";
        proxy_set_header X-Forwarded-Proto "$scheme";
        proxy_set_header X-Forwarded-Host "$host";
        proxy_set_header X-Forwarded-Port "$server_port";
        proxy_set_header Upgrade "$http_upgrade";
        proxy_set_header Connection "$connection_upgrade";
        proxy_pass http://argocd_argocd-server_80$request_uri;
    }
}

But should instead always hit the default_server block you modified:

# FIXED: Default server block now has SSL cert and returns 444 instead of ssl_reject_handshake
server {
    listen 443 ssl default_server;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_nginx-gateway_bootstrap-ingress-gateway-cert.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_nginx-gateway_bootstrap-ingress-gateway-cert.pem;
    listen [::]:443 ssl default_server;
    return 444;
}

Since you modified it to include the ssl cert/key and return 444, the tls handshake will work and pass, however traffic won't get proxied to any upstream. It'll just drop it there.

If, in the case some traffic does make it to the upstream, that means it must have somehow had the server_name argocd-emeka.dragonflyft.com;, meaning SNI wasn't dropped. And the original nginx conf:

server {
    listen 443 ssl default_server;
    listen [::]:443 ssl default_server;
    ssl_reject_handshake on;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_nginx-gateway_bootstrap-ingress-gateway-cert.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_nginx-gateway_bootstrap-ingress-gateway-cert.pem;

    server_name argocd.example.com;


    location / {






        proxy_http_version 1.1;
        proxy_set_header Host "$gw_api_compliant_host";
        proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for";
        proxy_set_header X-Real-IP "$remote_addr";
        proxy_set_header X-Forwarded-Proto "$scheme";
        proxy_set_header X-Forwarded-Host "$host";
        proxy_set_header X-Forwarded-Port "$server_port";
        proxy_set_header Upgrade "$http_upgrade";
        proxy_set_header Connection "$connection_upgrade";
        proxy_pass http://argocd_argocd-server_80$request_uri;



    }
}

Should work since it won't go into the default_server block.

With your modified nginx conf, are you seeing traffic reach your argocd backend?