[CI]: content digest not found follow-up

[apostasie]

Description

Following the patch changing async image delete to sync and avoiding Convert problematic codepath, content digest not found all but disappeared EXCEPT on TestRemove/Remove_image_with_running_container, which does surface it when calling nerdctl images.

Intuition is that this might be showcasing the underlying containerd issue with garbage collection / deletion of images with (repeat) identical content, but this is an unproven hunch.

Steps to reproduce the issue

Repeat on the CI.

Describe the results you received and expected

=================================
        | Command:	/usr/local/bin/nerdctl images
        | Working Dir:	/tmp/TestRemoveRemove_image_with_running_container_-_with_-f27[956](https://github.com/containerd/nerdctl/actions/runs/14452145394/job/40527165424?pr=4103#step:7:957)83035/001
        | Timeout:	3m0s
        =================================
        	HOSTNAME=df0f0ac2e041
        	MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=
        	SYSTEMD_EXEC_PID=80
        	container=docker
        	HOME=/root
        	LANG=C.UTF-8
        	MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/system.slice/docker-entrypoint.service/memory.pressure
        	INVOCATION_ID=8ceae0c7056f47ea912930034432fc8e
        	TERM=xterm
        	USER=root
        	SHLVL=3
        	CGO_ENABLED=0
        	_=/usr/local/bin/gotestsum
        	PATH=/usr/local/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
        	***
        	DOCKER_CONFIG=/tmp/TestRemoveRemove_image_with_running_container_-_with_-f2795683035/001
        	NERDCTL_TOML=/tmp/TestRemoveRemove_image_with_running_container_-_with_-f2795683035/001/nerdctl.toml
        =================================
        | Stderr:
        =================================
        time="2025-04-14T17:51:45Z" level=warning msg="content digest sha256:ec14c7992a97fc11425907e908340c6c3d6ff602f5f13d899e6b7027c9b4133a: not found"
        
        =================================
        | Stdout:
        =================================
        REPOSITORY    TAG    IMAGE ID    CREATED    PLATFORM    SIZE    BLOB SIZE
        
        =================================
        | Exit Code: 0
        | Signaled: <nil>
        | Err: <nil>
        =================================
        	👀 testing:		`REPOSITORY    TAG    IMAGE ID    CREATED    PLATFORM    SIZE    BLOB SIZE
        `
        	❌ does not verify:	~= `<none>`

What version of nerdctl are you using?

main

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

No response

[apostasie]

Note the post-test cleanup failure:

 image_remove_test.go:136: 
        
        +------------------------------------------------------------------------------------------------------------+
        | ➡️      | ⚙️ /usr/local/bin/nerdctl rmi -f ec14c799                                                        |
        +------------------------------------------------------------------------------------------------------------+
        |         | 🟢 Untagged: :@sha256:ec14c7992a97fc11425907e908340c6c3d6ff602f5f13d899e6b7027c9b4133a           |
        +------------------------------------------------------------------------------------------------------------+
        |         | 🟠 time="2025-04-15T05:54:47Z" level=warning msg="failed to enumerate rootfs" error="content dig |
        |         | est sha256:ec14c7992a97fc11425907e908340c6c3d6ff602f5f13d899e6b7027c9b4133a: not found"          |
        +------------------------------------------------------------------------------------------------------------+
        | Limit   | 3m0s                                                                                             |
        +------------------------------------------------------------------------------------------------------------+
        | Environ | SHELL=/bin/bash                                                                                  |
        |         | LOGNAME=rootless                                                                                 |
        |         | XDG_SESSION_TYPE=tty                                                                             |
        |         | HOME=/home/rootless                                                                              |
        |         | LANG=C.UTF-8                                                                                     |
        |         | SSH_CONNECTION=::1 40560 ::1 22                                                                  |
        |         | XDG_SESSION_CLASS=user                                                                           |
        |         | IPFS_PATH=/home/rootless/.local/share/ipfs                                                       |
        |         | USER=rootless                                                                                    |
        |         | SHLVL=1                                                                                          |
        |         | XDG_SESSION_ID=2                                                                                 |
        |         | XDG_RUNTIME_DIR=/run/user/1001                                                                   |
        |         | SSH_CLIENT=::1 40560 22                                                                          |
        |         | DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1001/bus                                            |
        |         | OLDPWD=/home/rootless                                                                            |
        |         | _=/usr/local/bin/gotestsum                                                                       |
        |         | PATH=/usr/local/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin |
        |         | :/bin:/usr/games:/usr/local/games:/snap/bin                                                      |
        |         | ***                                      |
        |         | NERDCTL_TOML=/tmp/TestRemoveRemove_image_with_running_container_-_with_-f2042042999/001/nerdctl. |
        |         | toml                                                                                             |
        |         | DOCKER_CONFIG=/tmp/TestRemoveRemove_image_with_running_container_-_with_-f2042042999/001         |
        +------------------------------------------------------------------------------------------------------------+

Looks to me like a strong indication that containerd did remove layers that it should not have.

[apostasie]

This very clearly is a consequence of the general content digest not found patch from #4062

The silver lining here is that the patch did quash most manifestations of the issue, which seem to now only happen as part of this test.
Whether the patch did miss some async delete, or whether we are now seeing containerd underlying issue in a crisper way is unclear.