[CI]: OCI runtime start failed: cannot start a container that has stopped: unknown #4143
Description
Description
This is seen in the context of compose (again on #3303 which is meant to surface our hidden issues).
Specifically in the case of the attached trace, for TestIPFSCompNoBuild.
The message is coming from containerd runc shim, probably from runc itself.
It is unclear to me if that would be a runc/containerd bug, or if nerdctl should do something to catch the underlying problem earlier.
At the very least, I would say that the output from containerd is not helpful enough.
Casual reading of source turns up:
- runc: https://github.com/opencontainers/runc/blob/main/start.go#L47 - refuses to start a stopped container
- called from containerd shim: https://github.com/containerd/containerd/blob/main/cmd/containerd-shim-runc-v2/process/init.go#L263
- which appears to be called only from here: https://github.com/containerd/containerd/blob/main/cmd/containerd-shim-runc-v2/process/init_state.go#L79 - note this is a method of
createdState
This problem clearly goes beyond compose - as the actual failure is from a (simple) run (though it might be specific to IPFS).
Describe the results you received and expected
=== Failed
=== FAIL: cmd/nerdctl/ipfs TestIPFSCompNoBuild/with-stargz-no-byAddr (100.26s)
requirements.go:238:
+------------------------------------------------------------------------------------------------------------+
| ➡️ | ⚙️ /usr/local/bin/nerdctl info --format {{ json . }} |
+------------------------------------------------------------------------------------------------------------+
| | 🟢 {"ID":"424471f2-16a8-4116-b393-0d1844a51eac","Driver":"overlayfs","Plugins":{"Log":["fluentd" |
| | ,"journald","json-file","none","syslog"],"Storage":["native","overlayfs","stargz","soci"]},"Memo |
| | ryLimit":true,"SwapLimit":true,"CpuCfsPeriod":true,"CpuCfsQuota":true,"CPUShares":true,"CPUSet": |
| | true,"CPURealtime":false,"PidsLimit":true,"IPv4Forwarding":true,"BridgeNfIptables":true,"BridgeN |
| | fIp6tables":true,"OomKillDisable":false,"SystemTime":"2025-04-20T18:52:35.058013148Z","LoggingDr |
| | iver":"json-file","CgroupDriver":"systemd","CgroupVersion":"2","KernelVersion":"6.8.0-1026-azure |
| | ","OperatingSystem":"Ubuntu 22.04.5 LTS","OSType":"linux","Architecture":"x86_64","NCPU":4,"MemT |
| | otal":16766754816,"Name":"25f5fde766b2","ServerVersion":"v1.6.38","SecurityOptions":["name=appar |
| | mor","name=seccomp,profile=builtin","name=cgroupns"],"Warnings":null} |
+------------------------------------------------------------------------------------------------------------+
| 🌱 | HOME=/root |
| | PATH=/usr/local/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin |
| | :/bin |
| | DOCKER_CONFIG=/tmp/TestIPFSCompNoBuildwith-stargz-no-byAddr1401898977/003 |
| | IPFS_PATH=/root/.ipfs |
| | NERDCTL_TOML=/tmp/TestIPFSCompNoBuildwith-stargz-no-byAddr1401898977/003/nerdctl.toml |
+------------------------------------------------------------------------------------------------------------+
| ⏰ | <1s (limit: 3m0s) |
+------------------------------------------------------------------------------------------------------------+
| 📁 | /tmp/TestIPFSCompNoBuildwith-stargz-no-byAddr1401898977/002 |
+------------------------------------------------------------------------------------------------------------+
ipfs_compose_linux_test.go:82:
+============================================================================================================+
| 🧽 | "TestIPFSCompNoBuild/with-stargz-no-byAddr": initial cleanup |
+============================================================================================================+
ipfs_compose_linux_test.go:82:
+============================================================================================================+
| 🏗 | "TestIPFSCompNoBuild/with-stargz-no-byAddr": setup |
+============================================================================================================+
ipfs_registry_linux_test.go:42:
+------------------------------------------------------------------------------------------------------------+
| ➡️ | ⚙️ /usr/local/bin/nerdctl push ipfs://ghcr.io/stargz-containers/wordpress:5.7-org --estargz |
+------------------------------------------------------------------------------------------------------------+
| | 🟢 bafkreigaiu2nrrqoecyzhsb337uhhi33b6algujcoavdqpnrlnhrre3d2i |
+------------------------------------------------------------------------------------------------------------+
| | 🟠 time="2025-04-20T18:52:35Z" level=info msg="pushing image \"ghcr.io/stargz-containers/wordpre |
| | ss:5.7-org\" to IPFS" |
| | time="2025-04-20T18:52:35Z" level=info msg="ensuring image contents" |
| | time="2025-04-20T18:52:35Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:af3407e398fd2b07f9d0e1a5120d5f9f345b0db0b307f34d35168dfc05bcd8df" |
| | time="2025-04-20T18:52:35Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:52e2fe65b69535f27db7f3e7f398b100dd1ef87e8748664dbbc5058d7443b023" |
| | time="2025-04-20T18:52:35Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:a55e9118c609b2e1c57e2aab71ef6b0656d633eb2bb8b3312d46b50bb6f210ad" |
| | time="2025-04-20T18:52:35Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:6df179caefdbb59e0360cb9089a2dd9bb4eda99f208ae339ecec075c81c2909f" |
| | time="2025-04-20T18:52:35Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:6eb4cea0dcf8e18df99e9d5605da003d540dc09e0b67d9f90fa0422cd85a8441" |
| | time="2025-04-20T18:52:35Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:4b7164a9b897e1bdf853fcc9891ee01b2a871f55fb5d154936195a6560983ecc" |
| | time="2025-04-20T18:52:35Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:30779a8cc4303de5cf25c55aecea25ada277b5d12d21e3ea309e59fbc45cf951" |
| | time="2025-04-20T18:52:35Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:13fd0cb07d3362bccaf9f79ab8490cba767c6397d4337ecdce84a2f67d05ab04" |
| | time="2025-04-20T18:52:35Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:0ab29b0dd494605f707a64557cba57bf18ca40e2dc2cdc78c5aff1912ca9d85a" |
| | time="2025-04-20T18:52:35Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:58985d1aa7515ef73f3b397d14b6bc520d0e6e27ad02c5dc7be723ff2312212c" |
| | time="2025-04-20T18:52:35Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:f6da7973abe819d8918d4d0677e117918885967f587471b5e9b4350c91e7c52a" |
| | time="2025-04-20T18:52:35Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:b9b327f2f6a5df8f83c1f825692a6cb9739bbbe5418e6efdb8bfc2b2b1609976" |
| | time="2025-04-20T18:52:35Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:1d79eaf46132ffd12dde835e55f9c183d68c59a9925a51b9416966d3454d8f9e" |
| | time="2025-04-20T18:52:41Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:7a4b6888c82c142209853b64fb0b4af22d781cce24a19d6425960407f025acb2" |
| | time="2025-04-20T18:52:54Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:d287a928953ae42fc68ffe270ab3aba6729e1155f455393beff735392b59637d" |
| | time="2025-04-20T18:52:59Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:792e2df4f0d9130ec0511f70222615f352b3e15dff1c0fa8e8fb199c8723721c" |
| | time="2025-04-20T18:53:00Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:3335058798708500df5262fc290b03e72c2e7ed8b88180ee714aff73012839fa" |
| | time="2025-04-20T18:53:00Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:c1ecd2ffe046900a34718d535d2b3241921fbbf5a97aadc549f1cc2cd300767d" |
| | time="2025-04-20T18:53:02Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:801305cbca6d0db7e979a3de8be96894fc54da48130086e0caf373940f4ad99e" |
| | time="2025-04-20T18:53:04Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:da675fec62a52728f35415495db6948434b183dd8ad02bf42d333a0389be2049" |
| | time="2025-04-20T18:53:26Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:802a64cac10561b11b45726ef8c7959108b30a7b0df1583ae04cf1ca3a6e4955" |
+------------------------------------------------------------------------------------------------------------+
| 🌱 | HOME=/root |
| | PATH=/usr/local/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin |
| | :/bin |
| | DOCKER_CONFIG=/tmp/TestIPFSCompNoBuildwith-stargz-no-byAddr1401898977/003 |
| | CONTAINERD_SNAPSHOTTER=stargz |
| | IPFS_PATH=/root/.ipfs |
| | NERDCTL_TOML=/tmp/TestIPFSCompNoBuildwith-stargz-no-byAddr1401898977/003/nerdctl.toml |
+------------------------------------------------------------------------------------------------------------+
| ⏰ | 52.230930444s (limit: 3m0s) |
+------------------------------------------------------------------------------------------------------------+
| 📁 | /tmp/TestIPFSCompNoBuildwith-stargz-no-byAddr1401898977/002 |
+------------------------------------------------------------------------------------------------------------+
ipfs_registry_linux_test.go:42:
+------------------------------------------------------------------------------------------------------------+
| ➡️ | ⚙️ /usr/local/bin/nerdctl push ipfs://ghcr.io/stargz-containers/mariadb:10.5-org --estargz |
+------------------------------------------------------------------------------------------------------------+
| | 🟢 bafkreifcavuavdgei5acxx32lze6y5makzolurtsqzuqlmes6ljs6hpepy |
+------------------------------------------------------------------------------------------------------------+
| | 🟠 time="2025-04-20T18:53:27Z" level=info msg="pushing image \"ghcr.io/stargz-containers/mariadb |
| | :10.5-org\" to IPFS" |
| | time="2025-04-20T18:53:27Z" level=info msg="ensuring image contents" |
| | time="2025-04-20T18:53:27Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:dc48f8554e3ecf2c48e78d5ea02d2264cf9917121403125008655d7925761f26" |
| | time="2025-04-20T18:53:27Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:a39a38ac740785e42963e064aab4a826bedfea11e1b6fc0bdf1d37a2d57d8c04" |
| | time="2025-04-20T18:53:27Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:d239b135d6d98baf244da83ef4ffaf9b20531ff9a9b2fe4a310d6cae15de7b24" |
| | time="2025-04-20T18:53:27Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:67dc80b40efe0f6719857dc7139e22eedf76523f12ed3d913bc0834495327568" |
| | time="2025-04-20T18:53:27Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:199841dea9ee64dc7e3752d60f46d13318a761944ee2e7ea1978af1cec402981" |
| | time="2025-04-20T18:53:27Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:9b9f86d4d9f02ce721704513f91e5c4a5760148985392dcfb0e8345213de8305" |
| | time="2025-04-20T18:53:27Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:10ca8445ec437675a903ff1676d9e763ec1114546f297bc86714d8530df8dcdd" |
| | time="2025-04-20T18:53:29Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:aa5e8b7bff97c8f0a07d828b8b62fed6404992f9b12a9327d35c17c22ae4d1c3" |
| | time="2025-04-20T18:53:29Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:49ddfbd153265f6979a72f8ca60fb3892c940fc534dd7f802b031fd3d786e937" |
| | time="2025-04-20T18:53:30Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:954cc10a5ec201c1d8a826b14d67d5576ca90017d90867823d017ee7d7a8f6eb" |
| | time="2025-04-20T18:53:39Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:dd48381931b3855afc8210799494e092ef72e287753ce9f6398ffb20dbd81d99" |
| | time="2025-04-20T18:54:12Z" level=info msg="converted \"application/vnd.docker.image.rootfs.diff |
| | .tar.gzip\" to sha256:d40e6590d91bdb54d2d127466d7e3973b41f2ee0c89a2b5eb66e7335c33efb86" |
+------------------------------------------------------------------------------------------------------------+
| 🌱 | HOME=/root |
| | PATH=/usr/local/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin |
| | :/bin |
| | NERDCTL_TOML=/tmp/TestIPFSCompNoBuildwith-stargz-no-byAddr1401898977/003/nerdctl.toml |
| | DOCKER_CONFIG=/tmp/TestIPFSCompNoBuildwith-stargz-no-byAddr1401898977/003 |
| | CONTAINERD_SNAPSHOTTER=stargz |
| | IPFS_PATH=/root/.ipfs |
+------------------------------------------------------------------------------------------------------------+
| ⏰ | 46.196496613s (limit: 3m0s) |
+------------------------------------------------------------------------------------------------------------+
| 📁 | /tmp/TestIPFSCompNoBuildwith-stargz-no-byAddr1401898977/002 |
+------------------------------------------------------------------------------------------------------------+
ipfs_compose_linux_test.go:276:
+------------------------------------------------------------------------------------------------------------+
| ➡️ | ⚙️ /usr/local/bin/nerdctl compose -f /tmp/nerdctl-compose-test[2257](https://github.com/containerd/nerdctl/actions/runs/14562176627/job/40846937175?pr=4140#step:8:2258)488992/docker-compose.yaml dow |
| | n -v |
+------------------------------------------------------------------------------------------------------------+
| | 🟠 time="2025-04-20T18:54:13Z" level=warning msg="/tmp/nerdctl-compose-test2257488992/docker-com |
| | pose.yaml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid po |
| | tential confusion" |
+------------------------------------------------------------------------------------------------------------+
| 🌱 | HOME=/root |
| | PATH=/usr/local/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin |
| | :/bin |
| | DOCKER_CONFIG=/tmp/TestIPFSCompNoBuildwith-stargz-no-byAddr1401898977/003 |
| | CONTAINERD_SNAPSHOTTER=stargz |
| | IPFS_PATH=/root/.ipfs |
| | NERDCTL_TOML=/tmp/TestIPFSCompNoBuildwith-stargz-no-byAddr1401898977/003/nerdctl.toml |
+------------------------------------------------------------------------------------------------------------+
| ⏰ | <1s (limit: 3m0s) |
+------------------------------------------------------------------------------------------------------------+
| 📁 | /tmp/TestIPFSCompNoBuildwith-stargz-no-byAddr1401898977/002 |
+------------------------------------------------------------------------------------------------------------+
ipfs_compose_linux_test.go:286:
+------------------------------------------------------------------------------------------------------------+
| ➡️ | ⚙️ /usr/local/bin/nerdctl compose -f /tmp/nerdctl-compose-test2257488992/docker-compose.yaml up |
| | --quiet-pull -d |
+------------------------------------------------------------------------------------------------------------+
| | 🚫 command returned a non-zero exit code |
+------------------------------------------------------------------------------------------------------------+
| | ⚠️ 1 |
+------------------------------------------------------------------------------------------------------------+
| | 🟠 time="2025-04-20T18:54:13Z" level=warning msg="/tmp/nerdctl-compose-test2257488992/docker-com |
| | pose.yaml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid po |
| | tential confusion" |
| | time="2025-04-20T18:54:13Z" level=info msg="Creating network nerdctl-compose-test2257488992_defa |
| | ult" |
| | time="2025-04-20T18:54:13Z" level=info msg="Creating volume nerdctl-compose-test2257488992_wordp |
| | ress" |
| | time="2025-04-20T18:54:13Z" level=info msg="Creating volume nerdctl-compose-test2257488992_db" |
| | time="2025-04-20T18:54:13Z" level=info msg="Ensuring image ipfs://bafkreigaiu2nrrqoecyzhsb337uhh |
| | i33b6algujcoavdqpnrlnhrre3d2i" |
| | time="2025-04-20T18:54:13Z" level=info msg="Ensuring image ipfs://bafkreifcavuavdgei5acxx32lze6y |
| | 5makzolurtsqzuqlmes6ljs6hpepy" |
| | time="2025-04-20T18:54:13Z" level=info msg="Creating container nerdctl-compose-test2257488992-wo |
| | rdpress-1" |
| | time="2025-04-20T18:54:13Z" level=info msg="Running [/usr/local/bin/nerdctl --namespace=nerdctl- |
| | test run --cidfile=/tmp/compose-1443933266/cid -l=com.docker.compose.project=nerdctl-compose-tes |
| | t2257488992 -l=com.docker.compose.service=wordpress -d --name=nerdctl-compose-test2257488992-wor |
| | dpress-1 --pull=never -e=WORDPRESS_DB_USER=exampleuser -e=WORDPRESS_DB_PASSWORD=examplepass -e=W |
| | ORDPRESS_DB_NAME=exampledb -e=WORDPRESS_DB_HOST=db --net=nerdctl-compose-test2257488992_default |
| | --hostname=wordpress -p=5004:80/tcp --restart=always -v=nerdctl-compose-test2257488992_wordpress |
| | :/var/www/html ipfs://bafkreigaiu2nrrqoecyzhsb337uhhi33b6algujcoavdqpnrlnhrre3d2i]" |
| | time="2025-04-20T18:54:13Z" level=warning msg="volume \"nerdctl-compose-test2257488992_wordpress |
| | \" already exists and will be returned as-is" |
| | time="2025-04-20T18:54:14Z" level=fatal msg="OCI runtime start failed: cannot start a container |
| | that has stopped: unknown" |
| | time="2025-04-20T18:54:14Z" level=fatal msg="error while creating container nerdctl-compose-test |
| | 2257488992-wordpress-1: error while creating container nerdctl-compose-test2257488992-wordpress- |
| | 1: exit status 1" |
+------------------------------------------------------------------------------------------------------------+
| 🌱 | HOME=/root |
| | PATH=/usr/local/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin |
| | :/bin |
| | IPFS_PATH=/root/.ipfs |
| | NERDCTL_TOML=/tmp/TestIPFSCompNoBuildwith-stargz-no-byAddr1401898977/003/nerdctl.toml |
| | DOCKER_CONFIG=/tmp/TestIPFSCompNoBuildwith-stargz-no-byAddr1401898977/003 |
| | CONTAINERD_SNAPSHOTTER=stargz |
+------------------------------------------------------------------------------------------------------------+
| ⏰ | <1s (limit: 3m0s) |
+------------------------------------------------------------------------------------------------------------+
| 📁 | /tmp/TestIPFSCompNoBuildwith-stargz-no-byAddr1401898977/002 |
+------------------------------------------------------------------------------------------------------------+
ipfs_compose_linux_test.go:286: 🔗
<<<<<<<<<<<<<<<<<<<<
🖊️ Command should succeed
👀 testing: `command returned a non-zero exit code`
❌ FAILED! is `<nil>`
>>>>>>>>>>>>>>>>>>>>
What version of nerdctl are you using?
main
Are you using a variant of nerdctl? (e.g., Rancher Desktop)
None
Host information
No response