[GR-42465] JFR jdk.ContainerConfiguration event reports wrong values in a container #5396
Description
Describe the issue
The jdk.ContainerConfiguration JFR event reports incorrect memory limits when run in a container. It reports unlimited values instead of the set container memory limit.
Steps to reproduce the issue
git clone https://github.com/Karm/mandrel-integration-tests.gitcd mandrel-integration-tests/apps/debug-symbols-smokemvn clean packagenative-image --enable-monitoring -jar target/debug-symbols-smoke.jar target/debug-symbols-smoke- Run the result in a container for example with
sudo podman run --memory=1G --memory-swap=1G --rm -ti -v $(pwd)/target:/opt/target:z -v /path/to/graalvm:/opt/jdk:z fedora:37 [root@62988404d0a9 opt]# /opt/target/debug-symbols-smoke -XX:+FlightRecorder -XX:StartFlightRecording=filename=flight-native.jfr -XX:FlightRecorderLogging=jfr- Inspect the JFR events for example with the
jfrtool:/opt/jdk/bin/jfr print --events jdk.ContainerConfiguration flight-native.jfr
Observe the memoryLimit and swapMemoryLimit being reported incorrectly. Shows -1, while it should be showing 1 GB.
Describe GraalVM and your environment:
- GraalVM version: Graal master, revision
7b1952da4031ae9f8446b2e04e6d9b8575607004 - JDK major version: 17
- OS: Linux
- Architecture: AMD64
More details
It's not reproducible with JVM mode:
[root@62988404d0a9 opt]# ./jdk/bin/java -XX:StartFlightRecording=filename=flight-java.jfr -Xlog:jfr -jar ./target/debug-symbols-smoke.jar
[0.301s][info][jfr] Flight Recorder initialized
[0.301s][info][jfr] Repository base directory: /tmp
[0.386s][info][jfr] Started recording "1" (1) {dumponexit=true, filename=/opt/flight-java.jfr}
[0.386s][info][jfr,startup] Started recording 1. No limit specified, using maxsize=250MB as default.
[0.386s][info][jfr,startup]
[0.386s][info][jfr,startup] Use jcmd 13 JFR.dump name=1 to copy recording data to file.
Q to quit
Q
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
[4.274s][info][jfr ] Stopped recording "1" (1). Reason "Dump on exit".
[4.276s][info][jfr ] Transferred 208454 bytes from the disk repository
[4.276s][info][jfr ] Wrote recording "1" (1) to /opt/flight-java.jfr
[4.276s][info][jfr ] Closed recording "1" (1)
[4.278s][info][jfr ] Removed repository /tmp/2022_11_09_12_53_49_13
[root@62988404d0a9 opt]# /opt/jdk/bin/jfr print --events jdk.ContainerConfiguration flight-java.jfr
jdk.ContainerConfiguration {
startTime = 12:53:49.374
containerType = "cgroupv1"
cpuSlicePeriod = 100 ms
cpuQuota = 200 ms
cpuShares = -1
effectiveCpuCount = 2
memorySoftLimit = -1 byte
memoryLimit = 1.0 GB
swapMemoryLimit = 1.0 GB
eventThread = "main" (javaThreadId = 1)
}
When debugging in gdb, it looks like build-time paths are being used for looking up the container limits:
$ gdb -ex 'set debuginfod enabled off' -ex 'set breakpoints pending on' -ex "break 'jdk.internal.platform.cgroupv1.CgroupV1Subsystem::getMemoryLimit'" -ex run --args ./target/debug-symbols-smoke -XX:+FlightRecorder -XX:StartFlightRecording=filename=flight-native.jfr -XX:FlightRecorderLogging=jfr
GNU gdb (GDB) Fedora 12.1-4.fc37
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./target/debug-symbols-smoke...
No symbol "breakpoints" in current context.
Breakpoint 1 at 0x86cfc0: file jdk/internal/platform/cgroupv1/CgroupV1Subsystem.java, line 310.
Starting program: /opt/target/debug-symbols-smoke -XX:+FlightRecorder -XX:StartFlightRecording=filename=flight-native.jfr -XX:FlightRecorderLogging=jfr
warning: Error disabling address space randomization: Function not implemented
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7f06d93ff6c0 (LWP 180)]
[New Thread 0x7f06d89ff6c0 (LWP 181)]
[New Thread 0x7f06d3dff6c0 (LWP 182)]
[info][jfr] Added periodic hook for EveryChunkPeriodEvents(10542)
[info][jfr] Added periodic hook for EndChunkPeriodEvents(10541)
[New Thread 0x7f06d2d726c0 (LWP 183)]
[New Thread 0x7f06d23ff6c0 (LWP 184)]
[info][jfr] Flight Recorder initialized
[info][jfr] Repository base directory: /tmp
Thread 1 "debug-symbols-s" hit Breakpoint 1, jdk.internal.platform.cgroupv1.CgroupV1Subsystem::getMemoryLimit() (this=0x7f06d9ce60e0) at jdk/internal/platform/cgroupv1/CgroupV1Subsystem.java:310
310 jdk/internal/platform/cgroupv1/CgroupV1Subsystem.java: No such file or directory.
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.36-7.fc37.x86_64 zlib-1.2.12-5.fc37.x86_64
(gdb) list
305 public long getMemoryFailCount() {
306 return getLongValue(memory, "memory.failcnt");
307 }
308
309 public long getMemoryLimit() {
310 long retval = getLongValue(memory, "memory.limit_in_bytes");
311 if (retval > CgroupV1SubsystemController.UNLIMITED_MIN) {
312 if (memory.isHierarchical()) {
313 // memory.limit_in_bytes returned unlimited, attempt
314 // hierarchical memory limit
(gdb) break getLongValue
Breakpoint 2 at 0x9eea50: getLongValue. (2 locations)
(gdb) c
Continuing.
Thread 1 "debug-symbols-s" hit Breakpoint 2, jdk.internal.platform.CgroupSubsystemController::getLongValue(jdk.internal.platform.CgroupSubsystemController*, java.lang.String*, java.util.function.Function*, long) (controller=0x7f06d9d79d48, param=0x7f06d9941580, conversion=0x7f06d9efe388, defaultRetval=-1) at jdk/internal/platform/CgroupSubsystemController.java:124
124 String strval = getStringValue(controller, param);
(gdb) step
jdk.internal.platform.CgroupUtil::readStringValue(jdk.internal.platform.CgroupSubsystemController*, java.lang.String*) (controller=0x7f06d9d79d48, param=0x7f06d9941580) at jdk/internal/platform/CgroupUtil.java:66
66 PrivilegedExceptionAction<BufferedReader> pea = () ->
(gdb) list
61 if (x instanceof Error)
62 throw (Error) x;
63 }
64
65 static String readStringValue(CgroupSubsystemController controller, String param) throws IOException {
66 PrivilegedExceptionAction<BufferedReader> pea = () ->
67 Files.newBufferedReader(Paths.get(controller.path(), param));
68 try (@SuppressWarnings("removal") BufferedReader bufferedReader =
69 AccessController.doPrivileged(pea)) {
70 String line = bufferedReader.readLine();
(gdb) break newBufferedReader
Breakpoint 3 at 0x635dd0: file java/nio/file/Files.java, line 2921.
(gdb) c
Continuing.
Thread 1 "debug-symbols-s" hit Breakpoint 3, java.nio.file.Files::newBufferedReader(java.nio.file.Path*, java.nio.charset.Charset*) (path=0x7f06d121ed80, cs=0x7f06d9d7b700) at java/nio/file/Files.java:2921
2921 CharsetDecoder decoder = cs.newDecoder();
(gdb) bt
#0 java.nio.file.Files::newBufferedReader(java.nio.file.Path*, java.nio.charset.Charset*) (path=0x7f06d121ed80, cs=0x7f06d9d7b700) at java/nio/file/Files.java:2921
#1 0x000000000086b9e9 in jdk.internal.platform.CgroupUtil::lambda$readStringValue$1(jdk.internal.platform.CgroupSubsystemController*, java.lang.String*) (controller=<optimized out>, param=<optimized out>)
at jdk/internal/platform/CgroupUtil.java:67
#2 0x000000000086b8e9 in jdk.internal.platform.CgroupUtil$$Lambda$b67b383100aa0509636893230ce25fa71f13b6c5::run() (this=<optimized out>)
#3 0x000000000063bfce in java.security.AccessController::executePrivileged(java.security.PrivilegedExceptionAction*, java.security.AccessControlContext*, java.lang.Class*) (action=<optimized out>, context=<optimized out>,
caller=<optimized out>) at com/oracle/svm/core/jdk/SecuritySubstitutions.java:147
#4 0x000000000063b779 in java.security.AccessController::doPrivileged(java.security.PrivilegedExceptionAction*) (action=<optimized out>) at java/security/AccessController.java:569
#5 0x000000000086c205 in jdk.internal.platform.CgroupUtil::readStringValue(jdk.internal.platform.CgroupSubsystemController*, java.lang.String*) (controller=<optimized out>, param=<optimized out>)
at jdk/internal/platform/CgroupUtil.java:69
#6 0x00000000009f7006 in jdk.internal.platform.CgroupSubsystemController::getLongValue(jdk.internal.platform.CgroupSubsystemController*, java.lang.String*, java.util.function.Function*, long) (controller=<optimized out>,
param=<optimized out>, conversion=0x7f06d9efe388, defaultRetval=-1) at jdk/internal/platform/CgroupSubsystemController.java:124
#7 0x000000000086d00c in jdk.internal.platform.cgroupv1.CgroupV1Subsystem::getMemoryLimit() (this=<optimized out>) at jdk/internal/platform/cgroupv1/CgroupV1Subsystem.java:310
#8 0x0000000000867124 in jdk.internal.platform.CgroupMetrics::getMemoryLimit() (this=<optimized out>) at jdk/internal/platform/CgroupMetrics.java:124
#9 0x0000000000917ec6 in jdk.jfr.internal.instrument.JDKEvents::emitContainerConfiguration() () at jdk/jfr/internal/instrument/JDKEvents.java:191
#10 0x0000000000917113 in jdk.jfr.internal.instrument.JDKEvents$$Lambda$068f755d65804fb86a1c5100fdb5369128bf04d6::run() (this=<optimized out>)
#11 0x00000000008cfcfb in jdk.jfr.internal.RequestEngine$RequestHook::execute() (this=0x7f06d945dbc0) at jdk/jfr/internal/RequestEngine.java:68
#12 0x00000000008d1073 in jdk.jfr.internal.RequestEngine::doChunk(java.util.function.Predicate*) (predicate=0x7f06d9efe508) at jdk/jfr/internal/RequestEngine.java:186
#13 0x00000000008c0720 in jdk.jfr.internal.PlatformRecorder::start(jdk.jfr.internal.PlatformRecording*) (this=<optimized out>, recording=<optimized out>) at jdk/jfr/internal/PlatformRecorder.java:287
#14 0x00000000008c89db in jdk.jfr.internal.PlatformRecording::start() (this=0x7f06d9467020) at jdk/jfr/internal/PlatformRecording.java:120
#15 0x000000000049affd in com.oracle.svm.core.jfr.JfrManager::initRecording() () at com/oracle/svm/core/jfr/JfrManager.java:246
#16 0x000000000049d4e6 in com.oracle.svm.core.jfr.JfrManager::lambda$startupHook$0(bool) (isFirstIsolate=<optimized out>) at com/oracle/svm/core/jfr/JfrManager.java:91
#17 0x0000000000499385 in com.oracle.svm.core.jfr.JfrManager$$Lambda$b3f652724e375272d1823cbd069b9505bf78a240::execute(bool) (this=<optimized out>, __0=<optimized out>)
#18 0x000000000048dbd7 in com.oracle.svm.core.jdk.RuntimeSupport::executeHooks(java.util.concurrent.atomic.AtomicReference*) (hooksReference=<optimized out>) at com/oracle/svm/core/jdk/RuntimeSupport.java:147
#19 0x000000000048dcec in com.oracle.svm.core.jdk.RuntimeSupport::initialize() (this=0x7f06d9e67c10) at com/oracle/svm/core/jdk/RuntimeSupport.java:92
#20 0x00000000004140ff in com.oracle.svm.core.JavaMainWrapper::runCore0() () at com/oracle/svm/core/JavaMainWrapper.java:161
#21 0x0000000000413ca5 in com.oracle.svm.core.JavaMainWrapper::doRun(int, org.graalvm.nativeimage.c.type.CCharPointerPointer*) (argc=<optimized out>, argv=<optimized out>) at com/oracle/svm/core/JavaMainWrapper.java:231
#22 0x000000000043c8fb in com.oracle.svm.core.code.IsolateEnterStub::JavaMainWrapper_run_e6899342f5939c89e6e2f78e2c71f5f4926b786d(int, org.graalvm.nativeimage.c.type.CCharPointerPointer*) (__0=<optimized out>, __1=<optimized out>)
at com/oracle/svm/core/code/IsolateEnterStub.java:1
(gdb) p path
$4 = (java.nio.file.Path *) 0x7f06d121ed80
[...]
(gdb) call path->toFile()
$7 = (java.io.File *) 0x7f06d121ee70
(gdb) call $7->toString()->value
$10 = (_z_.byte[] *) 0xfffffffff7c1edd8
(gdb) p (char*)$10
$11 = 0xfffffffff7c1edd8 <error: Cannot access memory at address 0xfffffffff7c1edd8>
(gdb) print *$7->toString()->value
$12 = {<byte []> = {<java.lang.Object> = {<_objhdr> = {hub = 0x4f2570 <com.oracle.svm.core.snippets.ImplicitExceptions::throwCachedClassCastException()+176>, idHash = 0}, <No data fields>}, len = 90,
data = 0x7f06d121ede8 "/sys/fs/cgroup/memory/user.slice/user-15263.slice/[email protected]/memory.limit_in_bytes"}, <No data fields>}
(gdb) shell cat /sys/fs/cgroup/memory/user.slice/user-15263.slice/[email protected]/memory.limit_in_bytes
cat: /sys/fs/cgroup/memory/user.slice/user-15263.slice/[email protected]/memory.limit_in_bytes: No such file or directory
(gdb) bt
#0 java.nio.file.Files::newBufferedReader(java.nio.file.Path*, java.nio.charset.Charset*) (path=0x7f06d121ed80, cs=0x7f06d9d7b700) at java/nio/file/Files.java:2921
#1 0x000000000086b9e9 in jdk.internal.platform.CgroupUtil::lambda$readStringValue$1(jdk.internal.platform.CgroupSubsystemController*, java.lang.String*) (controller=<optimized out>, param=<optimized out>)
at jdk/internal/platform/CgroupUtil.java:67
#2 0x000000000086b8e9 in jdk.internal.platform.CgroupUtil$$Lambda$b67b383100aa0509636893230ce25fa71f13b6c5::run() (this=<optimized out>)
#3 0x000000000063bfce in java.security.AccessController::executePrivileged(java.security.PrivilegedExceptionAction*, java.security.AccessControlContext*, java.lang.Class*) (action=<optimized out>, context=<optimized out>,
caller=<optimized out>) at com/oracle/svm/core/jdk/SecuritySubstitutions.java:147
#4 0x000000000063b779 in java.security.AccessController::doPrivileged(java.security.PrivilegedExceptionAction*) (action=<optimized out>) at java/security/AccessController.java:569
#5 0x000000000086c205 in jdk.internal.platform.CgroupUtil::readStringValue(jdk.internal.platform.CgroupSubsystemController*, java.lang.String*) (controller=<optimized out>, param=<optimized out>)
at jdk/internal/platform/CgroupUtil.java:69
#6 0x00000000009f7006 in jdk.internal.platform.CgroupSubsystemController::getLongValue(jdk.internal.platform.CgroupSubsystemController*, java.lang.String*, java.util.function.Function*, long) (controller=<optimized out>,
param=<optimized out>, conversion=0x7f06d9efe388, defaultRetval=-1) at jdk/internal/platform/CgroupSubsystemController.java:124
#7 0x000000000086d00c in jdk.internal.platform.cgroupv1.CgroupV1Subsystem::getMemoryLimit() (this=<optimized out>) at jdk/internal/platform/cgroupv1/CgroupV1Subsystem.java:310
#8 0x0000000000867124 in jdk.internal.platform.CgroupMetrics::getMemoryLimit() (this=<optimized out>) at jdk/internal/platform/CgroupMetrics.java:124
#9 0x0000000000917ec6 in jdk.jfr.internal.instrument.JDKEvents::emitContainerConfiguration() () at jdk/jfr/internal/instrument/JDKEvents.java:191
#10 0x0000000000917113 in jdk.jfr.internal.instrument.JDKEvents$$Lambda$068f755d65804fb86a1c5100fdb5369128bf04d6::run() (this=<optimized out>)
#11 0x00000000008cfcfb in jdk.jfr.internal.RequestEngine$RequestHook::execute() (this=0x7f06d945dbc0) at jdk/jfr/internal/RequestEngine.java:68
#12 0x00000000008d1073 in jdk.jfr.internal.RequestEngine::doChunk(java.util.function.Predicate*) (predicate=0x7f06d9efe508) at jdk/jfr/internal/RequestEngine.java:186
#13 0x00000000008c0720 in jdk.jfr.internal.PlatformRecorder::start(jdk.jfr.internal.PlatformRecording*) (this=<optimized out>, recording=<optimized out>) at jdk/jfr/internal/PlatformRecorder.java:287
#14 0x00000000008c89db in jdk.jfr.internal.PlatformRecording::start() (this=0x7f06d9467020) at jdk/jfr/internal/PlatformRecording.java:120
#15 0x000000000049affd in com.oracle.svm.core.jfr.JfrManager::initRecording() () at com/oracle/svm/core/jfr/JfrManager.java:246
#16 0x000000000049d4e6 in com.oracle.svm.core.jfr.JfrManager::lambda$startupHook$0(bool) (isFirstIsolate=<optimized out>) at com/oracle/svm/core/jfr/JfrManager.java:91
#17 0x0000000000499385 in com.oracle.svm.core.jfr.JfrManager$$Lambda$b3f652724e375272d1823cbd069b9505bf78a240::execute(bool) (this=<optimized out>, __0=<optimized out>)
#18 0x000000000048dbd7 in com.oracle.svm.core.jdk.RuntimeSupport::executeHooks(java.util.concurrent.atomic.AtomicReference*) (hooksReference=<optimized out>) at com/oracle/svm/core/jdk/RuntimeSupport.java:147
#19 0x000000000048dcec in com.oracle.svm.core.jdk.RuntimeSupport::initialize() (this=0x7f06d9e67c10) at com/oracle/svm/core/jdk/RuntimeSupport.java:92
#20 0x00000000004140ff in com.oracle.svm.core.JavaMainWrapper::runCore0() () at com/oracle/svm/core/JavaMainWrapper.java:161
#21 0x0000000000413ca5 in com.oracle.svm.core.JavaMainWrapper::doRun(int, org.graalvm.nativeimage.c.type.CCharPointerPointer*) (argc=<optimized out>, argv=<optimized out>) at com/oracle/svm/core/JavaMainWrapper.java:231
#22 0x000000000043c8fb in com.oracle.svm.core.code.IsolateEnterStub::JavaMainWrapper_run_e6899342f5939c89e6e2f78e2c71f5f4926b786d(int, org.graalvm.nativeimage.c.type.CCharPointerPointer*) (__0=<optimized out>, __1=<optimized out>)
at com/oracle/svm/core/code/IsolateEnterStub.java:1
(gdb)
Note the controller.path() value of "/sys/fs/cgroup/memory/user.slice/user-15263.slice/[email protected]/memory.limit_in_bytes" which is the build-host location, not the runtime location which should be /sys/fs/cgroup/memory/memory.limit_in_bytes:
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
1073741824
That correctly has the 1 GB value.