Starting container does not work if cwd is a symlink to another folder

[ekzhang]

Description

We noticed that if the cwd in an OCI image spec is a symlink to another folder in the rootfs, starting a container does not work. It appears to work in runc though.

{
  "ociVersion": "1.0.2-dev",
  // ...
  "cwd": "/this-folder-is-symlinked-to-another"
}

running container: starting container: starting root container: starting sandbox: failed to create process working directory "/eagle-data": not a directory

Context

I know this is bug report sounds strange(!) and so I just wanted to give some context as to why we ran into this edge case.

We support mounting "Volumes" in containers at Modal and creating sandboxes.

import modal

app = modal.App(name="eric-test-sandbox-workdir")


@app.local_entrypoint()
def main():
    sb = modal.Sandbox.create(
        "ls",
        "-a",
        app=app,
        volumes={
            "/eagle-data": modal.Volume.from_name("eagle-data"),
        },
        workdir="/eagle-data",
    )
    print(sb.stdout.read())
    sb.wait()
    print("Exit code:", sb.returncode)

We mount the volume in a path like /__modal/volumes/vo-123/..., and then symlink the user-provided path /eagle-data to that mount point. This is done so that we can commit / reload the volume dynamically when changes are made over the network, without interfering with user workloads.

But if we create a container with workdir set to /eagle-data, it fails to start up with this error message:

running container: starting container: starting root container: starting sandbox: failed to create process working directory "/eagle-data": not a directory

Error location

It appears to be triggered in this block of code.

gvisor/runsc/boot/vfs.go

Lines 205 to 211 in d839885

	if err := mntr.k.VFS().MkdirAllAt(
	ctx, procArgs.WorkingDirectory, mnsRoot, rootCreds,
	&vfs.MkdirOptions{Mode: 0755}, true, /* mustBeDir */
	); err != nil {
	return fmt.Errorf("failed to create process working directory %q: %w",
	procArgs.WorkingDirectory, err)
	}

And I think it could be fixed by adding FollowFinalSymlink: true to the flags of this PathOperation in vfs.go:

gvisor/pkg/sentry/vfs/vfs.go

Lines 875 to 881 in d839885

	func (vfs *VirtualFilesystem) MkdirAllAt(ctx context.Context, currentPath string, root VirtualDentry, creds *auth.Credentials, mkdirOpts *MkdirOptions, mustBeDir bool) error {
	pop := &PathOperation{
	Root: root,
	Start: root,
	Path: fspath.Parse(currentPath),
	}
	stat, err := vfs.StatAt(ctx, creds, pop, &StatOptions{Mask: linux.STATX_TYPE})

Docs of FollowFinalSymlink:

gvisor/pkg/sentry/vfs/vfs.go

Lines 233 to 236 in d839885

	// If FollowFinalSymlink is true, and the Dentry traversed by the final
	// path component represents a symbolic link, the symbolic link should be
	// followed.
	FollowFinalSymlink bool

Expected behavior

If we run the same container with runc, it works fine and the container starts up as usual. We have an internal ability to run this Modal reproduction with runc, and I verified with that.

Let us know if you need any more information or if we can help with a fix. Thanks!

Steps to reproduce

I've only reproduced it within Modal so far (sorry!) but I believe this should reproduce the issue in gVisor:

1. Create a rootfs for a container.
2. Create a symlink inside the rootfs by running sudo mkdir /foo and sudo ln -s /foo /bar, this points /bar -> /foo.
3. Start gVisor with "cwd": "/bar".

runsc version

runsc version bb08e9604675
spec: 1.2.0

docker version (if using docker)

Not using docker

uname

Linux ip-10-110-33-234.sa-east-1.compute.internal 5.15.0-309.180.4.el9uek.x86_64 #2 SMP Wed May 21 06:56:22 PDT 2025 x86_64 x86_64 x86_64 GNU/Linux

kubectl (if using Kubernetes)

Not using Kubernetes

repo state (if built from source)

Clean state on the commit sha

runsc debug logs (if available)

[cweld510]

We think that this behavior broke in 8a1a040. Prior to this change, gvisor could follow symlinks for the cwd.

[ayushr2]

cc @avagin