nerdctl push command not working when using minikube with containerd (`invalid checksum digest format`)

[mehuaniket]

Description

I currently use the nerdctl tool and a minikube cluster. I have a specific use case where I must commit an image and push it to a Docker Hub registry. However, every time I attempt to do this, I encounter the following error:

FATA[0001] failed commit on ref "index-sha256:8ee9676e710295f2ed187f72de413b4fe71e00587b4b069ae3e12777b7b9f01d": invalid content digest in response: invalid checksum digest format
I attempted to pull and push official docker images, tagged them with my personal account but was unable to push them to my repository.

Steps to reproduce the issue

1. I installed minikube cluster with cotnainerd

minikube start --ports=443 --ports=80 --container-runtime=containerd --cni=cilium --kubernetes-version=1.27

2. Installed nerdctl

wget https://github.com/containerd/nerdctl/releases/download/v1.5.0/nerdctl-1.5.0-linux-amd64.tar.gz
tar -xvf nerdctl-1.5.0-linux-amd64.tar.gz
mv nerdctl /usr/local/bin

3. sudo nerdctl login
4. sudo nerdctl pull
5. sudo nerdctl tag <private_image> <private_image_with_new_tag>
6. sudo nerdctl push <private_image_with_new_tag>

Describe the results you received and expected

The result I received:

initial error:

docker@minikube:~$ sudo nerdctl -n k8s.io push USERNAME/nginx_nerdctl --all-platforms --debug
DEBU[0000] Found hosts dir "/etc/containerd/certs.d"
DEBU[0000] Ignoring hosts dir "/etc/docker/certs.d" error="stat /etc/docker/certs.d: no such file or directory"
DEBU[0000] pushing digest="sha256:c2be4b8e4cdfsdsd71cb4e565f1bb83582d2841a0a8300509d079c92460734b11" image="docker.io/USERNAME/nginx_nerdctl:latest"
DEBU[0000] loading host directory dir=/etc/containerd/certs.d/docker.io
ERRO[0000] failed to decode hosts.toml error="invalid host tree"
manifest-sha256:c2be4b8e4cdf04f71cb4e565f1bb83582d2841a0a8300509d079c92460734b11: waiting |--------------------------------------|
elapsed: 0.1 s total: 0.0 B (0.0 B/s)
FATA[0000] content digest sha256:360eba32sdsda65016e0d558c6af176db31a202e9a6071666f9b629cb8ba6ccedf0: not found

I was able to resolve host and dir errors by creating dir and hosts.toml file.

Second:

FATA[0001] failed commit on ref "index-sha256:8ee9676e710295f2ed187f72de413b4fe71e00587b4b069ae3e12777b7b9f01d": invalid content digest in response: invalid checksum digest format

What version of nerdctl are you using?

1.5.0

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

Client:
Namespace: default
Debug Mode: false

Server:
Server Version: 1.6.21
Storage Driver: overlayfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Log: fluentd journald json-file syslog
Storage: aufs native overlayfs
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.15.0-82-generic
Operating System: Ubuntu 22.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 7
Total Memory: 6.001GiB
Name: minikube
ID: 1bef4b81-eeb8-4454-8c2f-f590c6267fd3

[AkihiroSuda]

sudo nerdctl -n k8s.io push USERNAME/nginx_nerdctl --all-platforms --debug --sign=cosign --snapshotter=native

What happens if you remove these additional flags?

[mehuaniket]

@AkihiroSuda It produces the same error. The flags resulted from my attempts to modify the output.

In my use case, I'm trying to commit and push the running container running on the minikube cluster. nerdctl commit worked fine, it is the push command that doesn't work.

[AkihiroSuda]

@AkihiroSuda It produces the same error. The flags resulted from my attempts to modify the output.

Please remove unneeded flags from the report.

[AkihiroSuda]

Also, is this issue really specific to minikube?

[mehuaniket]

I'm going to try the same thing with the rancher, I'll update the issue.

[mehuaniket]

I was able to perform similar steps successfully on the Rancher desktop cluster. I performed the steps in the following order.

1. curl -lo nginx.yaml "https://gist.githubusercontent.com/nonanom/498b913a69cede7037d55e28bb00344e/raw"
2. kubectl apply --filename nginx.yaml
3. rdctl shell
4. nerdctl -n k8s.io ps
5. nerdctl login
6. nerdctl commit b8766c83e050<nginx_container_id> mehuaniket/nginx:nerdctl_commit_push
7. nerdctl -n k8s.io push mehuaniket/nginx:nerdctl_commit_push

INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.v2+json, sha256:2854905c15d38618250bf18d226f078a7493e5a9d3dfaa5eb21231383fbff5b2) manifest-sha256:2854905c15d38618250bf18d226f078a7493e5a9d3dfaa5eb21231383fbff5b2: done |++++++++++++++++++++++++++++++++++++++| config-sha256:513f69a0953755e7a24b1fd6be97b6b1dce9cfbb2215b075d706606151c163a6: done |++++++++++++++++++++++++++++++++++++++| elapsed: 9.6 s total: 9.1 Ki (972.0 B/s)

[Zheaoli]

#827

seems the same issue