Cannot commit and push container image from Kubernetes pods

[nettoclaudio]

Hello folks! 👋

I'm experiencing a strange behavior while committing a container (from Kubernetes pod) and pushing it to some container registry after. I always get the same error from the push command: content digest sha256:898c46f3b1a1f39827ed135f020c32e2038c87ae0690a8fe73d94e5df9e6a2d6: not found.

I'm filing this issue here but there's a chance the problem is on containerd side because I've already seen it working in an earlier version. I'm trying to find out which version it stopped to work.

I'm still investigating but no idea so far... suggestions are welcome.

Steps to reproduce:

1. Create an arbitrary Kubernetes pod (for simplicity w/ just one container).

$ kubectl run --image tsuru/go:latest my-app -- sleep Inf

2. Execute arbitrary commands to generate changes in the container image (to be committed):

$ kubectl exec my-app -- mkdir -p /tmp/foo/bar

3. Commit the container to a local image:

$ CONTAINER_ID=$(kubectl get pods my-app -o 'jsonpath={ .status.containerStatuses[0].containerID }' | sed -E 's|(.+)://||g')
$ nerdctl --namespace k8s.io commit ${CONTAINER_ID} registry.example.com/my-app:v1

4. Push to local registry (no credentials needed):

$ nerdctl --namespace k8s.io push registry.example.com/my-app:v1
FATA[0000] failed to create a tmp single-platform image "registry.example.com/my-app:v1-tmp-reduced-platform": content digest sha256:898c46f3b1a1f39827ed135f020c32e2038c87ae0690a8fe73d94e5df9e6a2d6: not found

Environment information:

• nerdctl: 0.17.0
• containerd: v1.4.12 7b11cfaabd73bb80907dd23182b9347b4245eb5d
• OS: Buildroot 2021.02.4 (from minikube)
• Kernel: 4.19.202
• Kubernetes version: v1.20.6
• kubectl version: v1.20.6
• Minikube: v1.25.1

[junnplus]

I can't reproduce this problem on containerd with latest and 1.5.10 version.

jun@lima-k8s-test:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.6", GitCommit:"8a62859e515889f07e3e3be6a1080413f17cf2c3", GitTreeState:"clean", BuildDate:"2021-04-15T03:28:42Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.6", GitCommit:"8a62859e515889f07e3e3be6a1080413f17cf2c3", GitTreeState:"clean", BuildDate:"2021-04-15T03:19:55Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
jun@lima-k8s-test:~$ containerd -v
containerd containerd.io 1.5.10 2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc

In addition, I can't start the corresponding containerd version through minikube.

PS: containerd v1.4.x reached EOL.

[junnplus]

I closed it because containerd v1.4.x reached EOL.

Feel free to open it up if you have any other questions.

[Nirmit1995]

I am also getting the same error for large images (mine was 7GB, small images were easily pushed)

FATA[0000] failed to create a tmp single-platform image "registry.example.com/my-app:0086bf16-21cb-477b-952c-13780780f597-tmp-reduced-platform": content digest sha256:5bed26d33875e6da1d9ff9a1054c5fef3bbeb22ee979e14b72acf72528de007b: not found

I have the following config-
[ec2-user@ip]$ sudo /usr/local/bin/nerdctl version
WARN[0000] unable to determine buildctl version: exec: "buildctl": executable file not found in $PATH
Client:
Version: v1.4.0
OS/Arch: linux/amd64
Git commit: 7e8114a
buildctl:
Version:

Server:
containerd:
Version: 1.6.19
GitCommit: 1e1ea6e986c6c86565bc33d52e34b81b3e2bc71f
runc:
Version: 1.1.4
GitCommit: 5fd4c4d144137e991c4acebb2146ab1483a97925

Please help in resolving this.

[HFfleming]

could you please reopen the issue, @junnplus thanks.
I meet the same problem : I commit the image from a running container, then i need push the new image to my repo.
when i use nerdctl -n k8s.io push xxx ,an error occurs:
FATA[0000] failed to create a tmp single-platform image "xxx.com/hjmtest/nginx:hjm-tmp-reduced-platform": content digest sha256:550fe1bea624a5c62551cf09f3aa10886eed133794844af1dfb775118309387e: not found

here is my env info:
nerdctl -v nerdctl version 1.4.0
containerd -v containerd github.com/containerd/containerd v1.6.14-42-g21f32b2c3 21f32b2c394bc1ccfbf1744876a0fcdd4ef2390d

[HFfleming]

could you please open this issue, @junnplus thanks. I meet the same problem : I commit the image from a running container, then i need push the new image to my repo. when i use nerdctl -n k8s.io push xxx ,an error occurs: FATA[0000] failed to create a tmp single-platform image "xxx.com/hjmtest/nginx:hjm-tmp-reduced-platform": content digest sha256:550fe1bea624a5c62551cf09f3aa10886eed133794844af1dfb775118309387e: not found

here is my env info: nerdctl -v nerdctl version 1.4.0 containerd -v containerd github.com/containerd/containerd v1.6.14-42-g21f32b2c3 21f32b2c394bc1ccfbf1744876a0fcdd4ef2390d

In addition:
if i pull images from others, not commit a new image, nerdctl push xxx can work.

[ggerogery]

I got the same issue but with finch.
Solved with --platform arg:
finch push xxx.dkr.ecr.us-east-1.amazonaws.com/infra/calico/apiserver:v3.25.1 --platform linux/amd64

[baozaolaoba-top]

In addition:

nerdctl version 1.5.0
containerd -v
containerd github.com/containerd/containerd v1.7.1 1677a17964311325ed1c31e2c0a3589ce6d5c30d

harbor server with port: hub.local:5443

[Zheaoli]

WARN[0000] unable to determine buildctl version: exec: "buildctl": executable file not found in $PATH 
Client:
 Version:       v1.5.0
 OS/Arch:       linux/amd64
 Git commit:    b33a58f288bc42351404a016e694190b897cd252
 buildctl:
  Version:

Server:
 containerd:
  Version:      1.7.1+azure-1
  GitCommit:    1677a17964311325ed1c31e2c0a3589ce6d5c30d
 runc:
  Version:      1.1.7
  GitCommit:    860f061b76bb4fc671f0f9e900f7d80ff93d4eb7

could not produce this bug. would you guys mind to give me reproduce step with more detail? like what's registry you use

[JCereal]

Recently, my GKE cluster's worker node upgraded to v1.27.3-gke.100 which also upgraded containerd from 1.6.18 to 1.7.0. After the upgrade, I encountered the exact same issue.

Client:
 Version:	v1.7.0
 OS/Arch:	linux/amd64
 Git commit:	e674fe7ba6e49f12e88cd9c6c442e7ea5232502c
 buildctl:
  Version:	v0.12.3
  GitCommit:	438f47256f0decd64cc96084e22d3357da494c27

Server:
 containerd:
  Version:	1.7.0
  GitCommit:	1fbd70374134b891f97ce19c70b6e50c7b9f4e0d
 runc:
  Version:	1.1.10
  GitCommit:	v1.1.10-0-g18a0cb0f

Reproduce

The commit command: nerdctl -n k8s.io commit bbe68406a258 some.registry.com/my-app:0.1

The push command like: nerdctl -n k8s.io push some.registry.com/my-app:0.1

Output: FATA[0000] failed to create a tmp single-platform image "some.registry.com/my-app:0.1-tmp-reduced-platform": content digest sha256:c95ff01263cbbb536e71f8ae823d3e63f15f7a0f1ba9ecb7b3126a63654d2b23: not found.

Work around

Add --all-platforms option.

If I run the push command like: nerdctl -n k8s.io push --all-platforms some.registry.com/my-app:0.1, the image can be pushed if the registry is valid (btw I pushed to JFrog).

The option is added to avoid converter.Convert. Related code:

nerdctl/pkg/cmd/image/push.go

Lines 98 to 111 in ce2f63d

	if !options.AllPlatforms {
	pushRef = ref + "-tmp-reduced-platform"
	// Push fails with "400 Bad Request" when the manifest is multi-platform but we do not locally have multi-platform blobs.
	// So we create a tmp reduced-platform image to avoid the error.
	platImg, err := converter.Convert(ctx, client, pushRef, ref, converter.WithPlatform(platMC))
	if err != nil {
	if len(options.Platforms) == 0 {
	return fmt.Errorf("failed to create a tmp single-platform image %q: %w", pushRef, err)
	}
	return fmt.Errorf("failed to create a tmp reduced-platform image %q (platform=%v): %w", pushRef, options.Platforms, err)
	}
	defer client.ImageService().Delete(ctx, platImg.Name, images.SynchronousDelete())
	log.G(ctx).Infof("pushing as a reduced-platform image (%s, %s)", platImg.Target.MediaType, platImg.Target.Digest)
	}

Other info

The issue is not registry related. I tried different registries, the same error is thrown, even with random invalid registry.

The container I am committing from is built on an Ubuntu machine with docker build . command, without specifying the platform as linux/amd64.

When it used to work with containerd 1.6.18, I can see the log level=info msg="pushing as a reduced-platform image.... The convert should be good then.