Incorrectly pulls amd64 image on arm64 machine.

[narlark]

Describe the bug
On an arm machine when you pull the multi-arch image, it pulls the amd64 image instead of the arm64 image.

Steps to reproduce
Some weird behavior I'm noticing with upstream amazon/aws-node-termination-handler

On inspecting the manifest it's clearly a multi-arch image and does have an arm64 arch version available.

$ docker manifest inspect amazon/aws-node-termination-handler:v1.6.1
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 947,
         "digest": "sha256:7e91ba3ff76e3c540f8e2f1d3935b61fb05f831a9c36f42961a5d0e878e7c8a4",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 947,
         "digest": "sha256:f83038b59db9cebe1b1904fc6f351e82f8aa9e6cdb09bd89d34996998ad5ab16",
         "platform": {
            "architecture": "arm",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 947,
         "digest": "sha256:c3e1c2d17a05c8ecb661658bd179be3784fe605da58ab7940e1303c0fb8eda70",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1752,
         "digest": "sha256:2a72849bcd7c46518523072f8ad3453c451030a1ddfd2dd9c95bdae3ffdfdc8e",
         "platform": {
            "architecture": "amd64",
            "os": "windows",
            "os.version": "10.0.17763.1282"
         }
      }
   ]
}

But when I pull the image on my arm64 machine, it pulls the amd64 image instead of the arm64 image.

$ docker pull amazon/aws-node-termination-handler:v1.6.1

$ docker image inspect amazon/aws-node-termination-handler:v1.6.1
.....

            ],
            "OnBuild": null,
            "Labels": null
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 36635931,
        "VirtualSize": 36635931,
        "GraphDriver": {
....

Expected outcome
Architecture should be arm64 instead of amd64.

Application Logs
The log output when experiencing the issue.

Environment

• NTH App Version:
• NTH Mode (IMDS/Queue processor):
• OS/Arch: MacOS/arm64
• Kubernetes version:
• Installation method:

[snay2]

I reproduced the output you got on several different NTH versions (1.6.1, 1.7.0, 1.13.3, and 1.15.0) when pulling from both Docker Hub and ECR. The arm64 image does indeed think that it's amd64.

However, the arm64 image will run properly on an arm64 machine, even though docker issues a warning:

[ec2-user@ip-172-31-16-87 ~]$ uname -i
aarch64
[ec2-user@ip-172-31-16-87 ~]$ sudo docker run -it amazon/aws-node-termination-handler:v1.6.1-linux-arm64 --help
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Usage of /node-termination-handler:
  -cordon-only
...

The amd64 image fails in that scenario:

[ec2-user@ip-172-31-16-87 ~]$ uname -i
aarch64
[ec2-user@ip-172-31-16-87 ~]$ sudo docker run -it amazon/aws-node-termination-handler:v1.6.1-linux-amd64 --help
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
standard_init_linux.go:228: exec user process caused: exec format error

I also tested these same commands on an amd64 machine and verified that the amd64 image runs properly there while the arm64 image fails to run.

We will look into why the images incorrectly think they are amd64 even though they were built for (and run correctly on) arm64.

In the mean time, are you able to run NTH successfully on arm64 despite the architecture warning?

[narlark]

@snay2 Thanks for looking into it. At least the help cli command works without throwing errors. From the inspect it was clear that the image is an amd64 image and I just wanted to be sure no other functionality would fail further ahead because of this incorrect mapping.

[snay2]

It looks like the issue is the way our docker images get built. Even though the binaries are correctly built for each platform, the docker image base is always linux/amd64. (Probably similar to this issue.)

I am working on a solution that refactors our build system to use docker buildx build instead of its current configuration using a combination of docker build and docker manifest. Future releases after v1.15.0 will have that fix permanently on our ECR Public repo.

I can also do a one-off backfill of version v1.6.1 to both Docker Hub and ECR to resolve this mismatch for you.

[snay2]

I've updated the linux/arm64 and linux/amd64 builds of v1.6.1 on both Docker Hub and ECR Public. That should resolve the short term issue with that particular version.

PR incoming later today to prevent this issue in future builds.

[narlark]

@snay2 just pulled the latest build of v1.6.1 and it shows the correct architecture now when I do an inspect. Thanks a ton!

[narlark]

@snay2 Can you please backfill another version we are using as well? v1.12.1 on the ECR registry?

[snay2]

@narlark I've updated the linux/arm64 image for v1.12.1 on ECR and Docker Hub.

[narlark]

Verified the fix, thanks @snay2!