App hangs after taking crashdump in official .NET 9 x64 glibc docker images

[andrewlock]

Description

If a .NET 9 app running using the official .NET 9 glibc x64 images crashes and you take a crash dump, the process hangs indefinitely after writing the dump.

Note that AFAICT the issue only reproduces for the following tags, on x64

• 9.0.100-bookworm-slim
• 9.0.100-noble
• 9.0.100-azurelinux3.0

It does not reproduce on arm64, or on the following tags:

• 9.0.100-alpine3.20

The same issue is seen with both mcr.microsoft.com/dotnet/sdk and mcr.microsoft.com/dotnet/runtime

Reproduction Steps

Create a simple dockerfile as follows, that just throws an exception in Program.cs:

FROM mcr.microsoft.com/dotnet/sdk:9.0.100

WORKDIR /src

RUN mkdir /dumps \
    && dotnet new console -n SmokeTest -o . \
    && echo 'throw new System.Exception("Expected");' > Program.cs \
    && dotnet publish -c Release --framework net9.0 -o /src/publish

# Capture dumps
ENV COMPlus_DbgEnableMiniDump=1
ENV COMPlus_DbgMiniDumpType=4
ENV DOTNET_DbgMiniDumpName=/dumps/coredump.%t.%p

ENTRYPOINT ["dotnet", "/src/publish/SmokeTest.dll"]

You can build and run this using:

docker build -t smoketest .
docker run smoketest

Which produces the following output:

> docker run smoketest
Unhandled exception. System.Exception: Expected
   at Program.<Main>$(String[] args) in /src/Program.cs:line 1
[createdump] Gathering state for process 1 dotnet
[createdump] Crashing thread 0001 signal 6 (0006)
[createdump] Writing full dump to file /dumps/coredump.1731508490.1
[createdump] Written 146817024 bytes (35844 pages) to core file
[createdump] Target process is alive
[createdump] Dump successfully written in 132ms

but the process never ends, it is stuck at this point.

Note that I failed to reproduce this by running directly in Debian (in WSL). I could only reproduce it when running with the docker images

Expected behavior

The process should exit after createdump is called.

Actual behavior

The process does not exit.

docker exec into the process and running cat /proc/1/status shows it's frozen:

cat /proc/1/status
Name:   dotnet
Umask:  0022
State:  S (sleeping)

Regression?

This does not happen in .NET 8.

Known Workarounds

None, other than "don't collect a crash dump".

Configuration

This only reproduces with

• .NET 9 (both RC2 and GA)
• x64
• glibc (bookworm-slim, noble, azurelinux3.0)
• docker

Other information

@kevingosse noted this stack is strange, in that abort triggers the sigsegv handler instead of sigabrt:

#5  0x00007f83778e7125 in sigsegv_handler (code=11, siginfo=0x7f8376a9abf0, context=0x7f8376a9aac0)
    at /__w/1/s/src/coreclr/pal/src/exception/signal.cpp:677
#6  <signal handler called>
#7  __GI_abort () at ./stdlib/abort.c:107
#8  0x00007f8377912399 in PROCAbort (signal=<optimized out>, siginfo=siginfo@entry=0x0)
    at /__w/1/s/src/coreclr/pal/src/thread/process.cpp:2562

chatgpt suggests abort could raise a segfault if the sigabort information is invalid (and so a segfault is triggered when calling the sigabort handler), which sounds... plausible?

We hypothesized that this commit is the suspect, but mostly just because that's changed since .NET 8.

[dotnet-policy-service]

Tagging subscribers to this area: @tommcdon
See info in area-owners.md if you want to be subscribed.

[hoyosjs]

cc: @mikem8361 @dotnet/dotnet-diag

[mikem8361]

@janvorli do you think this has anything to do with PR #105578?

[janvorli]

@mikem8361 it is theoretically possible, although the SIGABORT should result in termination of the process, the fallback in the abort that uses SIGSEGV should never kick in. Let me debug the repro case.

[janvorli]

Without any testing done, I actually believe this happens due to the docker assigning PID=1 to the process. Linux handles signals differently for PID=1 as without docker, it is the init process. Such a process has special responsibilities. There is an article describing why running your own process as the only one in Docker is problematic: https://daveiscoding.hashnode.dev/why-do-you-need-an-init-process-inside-your-docker-container-pid-1

In the case of PID=1, the default action for Abort is "SIG_IGN". The abort() sends SIGABRT to the current process, but due to this, it is just ignored. So the abort() code falls back to killing the process using an instruction that triggers SIGSEGV. But since we don't uninstall the SIGSEGV handler after the PR #105578, this fallback doesn't terminate the process either.

I'll take a look if we can do something about it in our sigsegv handler. The workaround is to pass --init to docker when starting the container. That adds a "dummy" init process to the container and the actual .NET process would get PID=2 and everything would work as expected.

I've tried this workaround and it worked.

[janvorli]

I've attached debugger to the process to see if we can fix that problem. What happens is that the SIGSEGV triggered by the fallback path in abort() ends up calling PROCCreateCrashDump again. And it ends up spinning here:

runtime/src/coreclr/pal/src/thread/process.cpp

Lines 2235 to 2238 in 1e98c05

	while (true)
	{
	poll(NULL, 0, INFTIM);
	}

The reason is that the g_crashingThreadId still contains the current thread id that was set when the PROCCreateCrashDump was called.
@mikem8361 I think the fix would be to return with false if the previousThreadId == currentThreadId

[tommcdon]

fyi @dotnet/dotnet-monitor for awareness