[MAJOR] content digest not found

[apostasie]

This is meant to collect more information about the issue known as "content digest not found".

While #3435 (and follow-up for IPFS) did address the most obvious cases, we obviously have more.

Here is the currently know list:

• content digest not found: platform-reduced push, follow-up to #3425 #3509
• [CI]: content digest not found: IPFS not completely fixed #3510
• content digest not found issue is impacting nerdctl images --filter #3512

The gist of it appears to be, at this point:

• seems like containerd converter.Convert (calling on imageservice.Create) may return an image with missing content in some circumstances
• seems like any call to content.ReadBlob (specifically as seen inside imgutil) may fail and should be gated by an operation to first force fetch the blob in question

At this point, above is speculative, although it seems consistent with the debugging info in the tickets.

[apostasie]

Note that these bugs are elusive.
You have to run the test suite for the affected parts for 50+ times before you hit it.

[apostasie]

Update: this all likely ties into #3516

[apostasie]

Apparently happens as well on run:

This does reinforce the notion that there is a problem in containerd (possibly containerd/containerd#8973 )

https://github.com/containerd/nerdctl/actions/runs/11397513713/job/31713086655?pr=3535#step:7:538

Command:  /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 run --pull always -d --name testremove-remove-image-with-running-container-without-f-8c22042c ghcr.io/stargz-containers/alpine:3.13-org sleep infinity
        ExitCode: 1
        Error:    exit status 1
        Stdout:   
        Stderr:   ghcr.io/stargz-containers/alpine:3.13-org:                                     resolved       |++++++++++++++++++++++++++++++++++++++| 
        index-sha256:ec14c7992a97fc11425907e908340c6c3d6ff602f5f13d899e6b7027c9b4133a: done           |++++++++++++++++++++++++++++++++++++++| 
        elapsed: 0.1 s                                                                 total:  1.6 Ki (15.7 KiB/s)                                      
        ghcr.io/stargz-containers/alpine:3.13-org:                                        resolved       |++++++++++++++++++++++++++++++++++++++| 
        index-sha256:ec14c7992a97fc11425907e908340c6c3d6ff602f5f13d899e6b7027c9b4133a:    done           |++++++++++++++++++++++++++++++++++++++| 
        manifest-sha256:e103c1b4bf019dc290bcc7aca538dc2bf7a9d0fc836e186f5fa34945c5168310: done           |++++++++++++++++++++++++++++++++++++++| 
        config-sha256:49f356fa4513676c5e22e3a8404aad6c7262cc7aaed15341458265320786c58c:   done           |++++++++++++++++++++++++++++++++++++++| 
        elapsed: 0.2 s                                                                    total:  1.6 Ki (8.0 KiB/s)                                       
        time="2024-10-18T04:42:58Z" level=info msg="apply failure, attempting cleanup" error="failed to extract layer sha256:8ea3b23f387bedc5e3cee574742d748941443c328a75f511eb37b0d8b6164130: failed to get reader from content store: content digest sha256:ca3cd42a7c9525f6ce3d64c1a70982613a8235f0cc057ec9244052921853ef15: not found" key="extract-158142536-l19Q sha256:8ea3b23f387bedc5e3cee574742d748941443c328a75f511eb37b0d8b6164130"
        time="2024-10-18T04:42:58Z" level=fatal msg="error unpacking image: apply layer error for \"ghcr.io/stargz-containers/alpine:3.13-org\": failed to extract layer sha256:8ea3b23f387bedc5e3cee574742d748941443c328a75f511eb37b0d8b6164130: failed to get reader from content store: content digest sha256:ca3cd42a7c9525f6ce3d64c1a70982613a8235f0cc057ec9244052921853ef15: not found"
        
        Env:
        HOSTNAME=ff0daede016c
        MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=
        SYSTEMD_EXEC_PID=80
        container=docker
        HOME=/root
        LANG=C.UTF-8
        MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/system.slice/docker-entrypoint.service/memory.pressure
        INVOCATION_ID=1fb18e837[558](https://github.com/containerd/nerdctl/actions/runs/11397513713/job/31713086655?pr=3535#step:7:559)4703b789b82cf0a133cc
        TERM=xterm
        USER=root
        SHLVL=3
        CGO_ENABLED=0
        _=/usr/local/bin/gotestsum
        PATH=/usr/local/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
        ***
        DOCKER_CONFIG=/tmp/TestRemoveRemove_image_with_running_container_-_without_-f2863598993/001
        NERDCTL_TOML=/tmp/TestRemoveRemove_image_with_running_container_-_without_-f2863598993/001/nerdctl.toml
    case.go:164: ======================== Post-test cleanup ========================
    command.go:112: /usr/local/bin/nerdctl --namespace=testremove-private-a0b107b8 rm -f testremove-remove-image-with-running-container-without-f-8c22042c