GraalVM 24.0+ breaks the native-image driver (in native mode)

[jerboaa]

Description

After the move of GraalVM master (24.0) to use the JDK's metrics classes for container support, there is a cyclic dependency issue that needs to be solved for Mandrel.

1. PhysicalMemory class newly depends on the JDK's metrics classes, which use NIO in unpatched OpenJDK.
2. NIO, specifically Target_jdk_internal_misc_VM.java internally uses Runtime.getRuntime().maxMemory() for the initial direct memory size (overriden with -R:MaxDirectMemorySize), which is the heap size.
3. The implementation of Runtime.getRuntime().maxMemory() in HeapImpl.java uses PhysicalMemory.size().

Therefore, there is an initialization cycle resulting in a stack overflow. At runtime, it looks like this:

  i SP 0x00007ffdf838e070 IP 0x00000000008bf8b9 size=16 java.nio.file.Files.readAllLines(Files.java:3433)
  i SP 0x00007ffdf838e070 IP 0x00000000008bf8b9 size=16 jdk.internal.platform.CgroupUtil.lambda$readAllLinesPrivileged$2(CgroupUtil.java:83)
  A SP 0x00007ffdf838e070 IP 0x00000000008bf8b9 size=16 jdk.internal.platform.CgroupUtil$$Lambda$a0a20a05f23dafb9435259313f4041cb74d30494.run(Unknown Source)
  A SP 0x00007ffdf838e080 IP 0x000000000068283e size=32 java.security.AccessController.executePrivileged(AccessController.java:114)
  A SP 0x00007ffdf838e0a0 IP 0x00000000006822b9 size=32 java.security.AccessController.doPrivileged(AccessController.java:571)
  A SP 0x00007ffdf838e0c0 IP 0x00000000008bfb19 size=48 jdk.internal.platform.CgroupUtil.readAllLinesPrivileged(CgroupUtil.java:84)
  A SP 0x00007ffdf838e0f0 IP 0x00000000008bc715 size=176 jdk.internal.platform.CgroupSubsystemFactory.determineType(CgroupSubsystemFactory.java:143)
  A SP 0x00007ffdf838e1a0 IP 0x00000000008bbe2d size=16 jdk.internal.platform.CgroupSubsystemFactory.create(CgroupSubsystemFactory.java:85)
  i SP 0x00007ffdf838e1b0 IP 0x0000000000414cb3 size=16 jdk.internal.platform.CgroupMetrics.getInstance(CgroupMetrics.java:193)
  i SP 0x00007ffdf838e1b0 IP 0x0000000000414cb3 size=16 jdk.internal.platform.SystemMetrics.instance(SystemMetrics.java:29)
  i SP 0x00007ffdf838e1b0 IP 0x0000000000414cb3 size=16 jdk.internal.platform.Metrics.systemMetrics(Metrics.java:58)
  i SP 0x00007ffdf838e1b0 IP 0x0000000000414cb3 size=16 jdk.internal.platform.Container.metrics(Container.java:43)
  A SP 0x00007ffdf838e1b0 IP 0x0000000000414cb3 size=16 com.oracle.svm.core.Containers.memoryLimitInBytes(Containers.java:121)
  A SP 0x00007ffdf838e1c0 IP 0x0000000000475885 size=16 com.oracle.svm.core.heap.PhysicalMemory.size(PhysicalMemory.java:92)
  A SP 0x00007ffdf838e1d0 IP 0x0000000000565153 size=32 java.lang.Runtime.maxMemory(Runtime.java:932)
  A SP 0x00007ffdf838e1f0 IP 0x00000000004792d3 size=16 com.oracle.svm.core.jdk.DirectMemoryAccessors.initialize(Target_jdk_internal_misc_VM.java:112)
  i SP 0x00007ffdf838e200 IP 0x0000000000671b1a size=80 com.oracle.svm.core.jdk.DirectMemoryAccessors.getPageAlignDirectMemory(Target_jdk_internal_misc_VM.java:94)
  i SP 0x00007ffdf838e200 IP 0x0000000000671b1a size=80 jdk.internal.misc.VM.isDirectMemoryPageAligned(VM.java:156)
  A SP 0x00007ffdf838e200 IP 0x0000000000671b1a size=80 java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:124)
  i SP 0x00007ffdf838e250 IP 0x0000000000903265 size=48 java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:360)
  A SP 0x00007ffdf838e250 IP 0x0000000000903265 size=48 sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:242)
  A SP 0x00007ffdf838e280 IP 0x00000000008ff35e size=80 sun.nio.ch.IOUtil.read(IOUtil.java:303)
  i SP 0x00007ffdf838e2d0 IP 0x00000000008fda4d size=96 sun.nio.ch.IOUtil.read(IOUtil.java:283)
  A SP 0x00007ffdf838e2d0 IP 0x00000000008fda4d size=96 sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:234)
  i SP 0x00007ffdf838e330 IP 0x00000000008f8946 size=64 sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:74)
  A SP 0x00007ffdf838e330 IP 0x00000000008f8946 size=64 sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
  A SP 0x00007ffdf838e370 IP 0x000000000090601f size=64 sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:329)
  A SP 0x00007ffdf838e3b0 IP 0x00000000009052c3 size=48 sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:372)
  A SP 0x00007ffdf838e3e0 IP 0x0000000000905640 size=64 sun.nio.cs.StreamDecoder.lockedRead(StreamDecoder.java:215)
  A SP 0x00007ffdf838e420 IP 0x0000000000905ba9 size=64 sun.nio.cs.StreamDecoder.read(StreamDecoder.java:169)
  A SP 0x00007ffdf838e460 IP 0x00000000005281f5 size=16 java.io.InputStreamReader.read(InputStreamReader.java:188)
  A SP 0x00007ffdf838e470 IP 0x000000000051d331 size=48 java.io.BufferedReader.fill(BufferedReader.java:160)
  A SP 0x00007ffdf838e4a0 IP 0x000000000051d648 size=80 java.io.BufferedReader.implReadLine(BufferedReader.java:370) 

We need to find a way to set the initial direct memory size (without using NIO) and/or without creating this cycle. This isn't a problem in GraalVM Community since it uses labsjdk as a base, which has a patch avoiding this issue. Thus, it's a Mandrel only issue.

The issue can be reproduced when trying to generate the native-image generator as a native image itself on a recent mandrel build and then trying to run --version on the result:

$ native-image --macro:native-image-launcher
$ native-image --version

[jerboaa]

See also: oracle#7246 (comment) and https://bugs.openjdk.org/browse/JDK-8314940

[jerboaa]

Note that the macro builds with -H:-ParseRuntimeOptions. Thus runtime options don't seem to work to fix the stack overflow.

[jerboaa]

Draft fix for this. Needs more testing.

jerboaa/graal@no-jlinking-fixes-7085...jerboaa:graal:mandrel-cgroups-bb-dm

[jerboaa]

It looks like the cgroup change is a 24.0 only feature for now. It doesn't seem to affect 23.1 (current release branch as of today at fd442af), so I'm adjusting the milestone.

[jerboaa]

Draft fix for this. Needs more testing.

jerboaa/graal@no-jlinking-fixes-7085...jerboaa:graal:mandrel-cgroups-bb-dm

Test run with this patch including quarkus native tests:
https://github.com/graalvm/mandrel/actions/runs/6074785555