[Flaky] TestComposeExecWithIndex

[apostasie]

Description

Apparently, in certain circumstances, ips := helpers.Capture(append(cmds, "ip", "addr", "show", "dev", "eth0")...) fails to return meaningful data.

Comment says: // try 5 times to ensure that results are stable

So, this implies either:

• that after a compose up -d, a container might be started "enough" to be OK with an exec, but not ready enough for ip addr to return anything?
• or something is broken that makes it so the interface is not available in certain conditions

One way or the other, more debugging output is needed.

=== RUN   TestComposeExecWithIndex/1
=== PAUSE TestComposeExecWithIndex/1
=== CONT  TestComposeExecWithIndex/1
    compose_exec_linux_test.go:356: 
        
        +============================================================================================================+
        | 🏗       | "TestComposeExecWithIndex/1": setup                                                              |
        +============================================================================================================+
        
    compose_exec_linux_test.go:306: 
        
        +------------------------------------------------------------------------------------------------------------+
        | ➡️      | ⚙️ /usr/local/bin/nerdctl compose -f /tmp/TestComposeExecWithIndex4037248067/001/09b41bbf/compos |
        |         | e.yaml exec -i=false --no-TTY --index 1 svc0 cat /etc/hosts                                      |
        +------------------------------------------------------------------------------------------------------------+
        |         | 🟢 # <nerdctl>                                                                                   |
        |         | 127.0.0.1  localhost localhost.localdomain                                                       |
        |         | ::1    localhost localhost.localdomain                                                           |
        |         | 10.4.1.2        svc0 svc0.09b41bbf_default 09b41bbf-svc0-2 09b41bbf-svc0-2.09b41bbf_default      |
        |         | 10.4.1.4        svc0 svc0.09b41bbf_default 09b41bbf-svc0-1 09b41bbf-svc0-1.09b41bbf_default      |
        |         | 10.4.1.3        svc0 svc0.09b41bbf_default 09b41bbf-svc0-3 09b41bbf-svc0-3.09b41bbf_default      |
        |         | # </nerdctl>                                                                                     |
        +------------------------------------------------------------------------------------------------------------+
        | 🌱      | XDG_SESSION_TYPE=tty                                                                             |
        |         | HOME=/home/rootless                                                                              |
        |         | XDG_SESSION_CLASS=user                                                                           |
        |         | XDG_SESSION_ID=4                                                                                 |
        |         | XDG_RUNTIME_DIR=/run/user/1001                                                                   |
        |         | PATH=/usr/local/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin |
        |         | :/bin:/usr/games:/usr/local/games:/snap/bin                                                      |
        |         | DOCKER_CONFIG=/tmp/TestComposeExecWithIndex1237365141/003                                        |
        |         | NERDCTL_TOML=/tmp/TestComposeExecWithIndex1237365141/003/nerdctl.toml                            |
        +------------------------------------------------------------------------------------------------------------+
        | ⏰      | <1s (limit: 3m0s)                                                                                |
        +------------------------------------------------------------------------------------------------------------+
        | 📁      | /tmp/TestComposeExecWithIndex1237365141/002                                                      |
        +------------------------------------------------------------------------------------------------------------+
        
    compose_exec_linux_test.go:307: 
        
        +------------------------------------------------------------------------------------------------------------+
        | ➡️      | ⚙️ /usr/local/bin/nerdctl compose -f /tmp/TestComposeExecWithIndex4037248067/001/09b41bbf/compos |
        |         | e.yaml exec -i=false --no-TTY --index 1 svc0 ip addr show dev eth0                               |
        +------------------------------------------------------------------------------------------------------------+
        | 🌱      | XDG_SESSION_TYPE=tty                                                                             |
        |         | HOME=/home/rootless                                                                              |
        |         | XDG_SESSION_CLASS=user                                                                           |
        |         | XDG_SESSION_ID=4                                                                                 |
        |         | XDG_RUNTIME_DIR=/run/user/1001                                                                   |
        |         | PATH=/usr/local/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin |
        |         | :/bin:/usr/games:/usr/local/games:/snap/bin                                                      |
        |         | DOCKER_CONFIG=/tmp/TestComposeExecWithIndex1237365141/003                                        |
        |         | NERDCTL_TOML=/tmp/TestComposeExecWithIndex1237365141/003/nerdctl.toml                            |
        +------------------------------------------------------------------------------------------------------------+
        | ⏰      | 20.7[536](https://github.com/containerd/nerdctl/actions/runs/15885753263/job/44797144400?pr=4372#step:10:537)79825s (limit: 3m0s)                                                                      |
        +------------------------------------------------------------------------------------------------------------+
        | 📁      | /tmp/TestComposeExecWithIndex1237365141/002                                                      |
        +------------------------------------------------------------------------------------------------------------+
        
    compose_exec_linux_test.go:347: assertion failed: pip is nil: fail to get the real ip address
--- FAIL: TestComposeExecWithIndex/1 (21.62s)

Steps to reproduce the issue

Describe the results you received and expected

not fail

What version of nerdctl are you using?

main

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

No response

[apostasie]

Note: this also applies to other commands (cat /etc/hosts).

Failure appearing seems to be an unintended consequence of #4372 which now allows concurrent execs to happen causing the test timing to be much faster.

This specific test does spin up three replicas, and the exec in each replica are now executing in parallel.

[apostasie]

Walking out on a limb here - compose specialists please lecture me right if I am not.

Because we start the compose stack with -d, by the time we start execing into container A, container B might still be starting.
It will then update /etc/hosts, effectively blanking out the file for the already started containers before filling it again.
So, reading /etc/hosts in container A might not be reliable until all containers have started.
If this is right, we probably have the same problem for all containers producing a similar scenario (eg: two containers starting in parallel on the same network).

I am not sure there is a good solution for this.
Maybe we could better split /etc/hosts so that we only change the "other hosts" section when we need to update? But then again, it won't work in all cases.

About the fact that ip addr also returns nil, I do not know - though this is possibly the same root cause?