Native Memory Tracking and Associated JFR events

[roberttoyonaga]

Feature request

Is your feature request related to a problem? Please describe.
Not related to a problem.

Describe the solution you'd like.
Hotspot JFR has native memory tracking (NMT) events. It may be useful to have support for NMT in SubstrateVM.

In Hotspot, NMT hooks into malloc and also the virtual memory management code (where mmap is used). This allows for tracking how much memory each VM component needs for its own operations. Native memory is grouped into categories (ie GC, Metaspace, etc). The JFR NMT events are just a small feature built on top of the already existing NMT infrastructure.

In SubstrateVM, the memory used by the java application and VM itself is not so clearly separated because memory used by the VM can be managed by the GC. So I can think of two approaches:

1. Record only off-heap memory. Perhaps adding tracking in places such as org.graalvm.nativeimage.UnmanagedMemory#malloc
   • This wouldn't provide a complete picture of how much memory each VM component is actually using. It would at least fill in the gaps left by heap dumps and JFR allocation events.
2. Determine a way to distinguish VM-level allocations from application level ones. This would provide a more complete breakdown of what memory is needed for VM internal operations. I think this is probably more complicated to accomplish.

In SubstrateVM there are three areas where the VM itself might contribute to memory usage: the java heap, off-heap, and the image heap.

To implement NMT in Native Image we first need to decide what "native memory" we want to track. Does "native memory" mean all memory used by the VM for its internal operations, or does it simply mean unmanaged memory? In the Hotspost NMT implementation, these are basically the same thing -- memory used by the VM is not managed by the GC.

In Hotspot, there are two NMT modes "summary" and "detailed". The "detailed" mode includes call site information, whereas the "summary" only provides a memory amounts per category. As a first step, I think the goal should be scoped down to only the "summary" mode.

Describe who do you think will benefit the most.
GraalVM users and contributors would benefit. Some use cases would be:

1. RSS is higher than expected or exceeds specified heap size. NMT can shed light on memory usage contributing to RSS not visible in heap dumps or JFR allocation events.
2. A user or GraalVM developer may want to measure how much each VM component contribues to RSS. For example, is JFR too expensive to use in their native image executables for their use-case?

Express whether you'd like to help contributing this feature
I'd like to help contribute this if it is desired and feasible.

[christianhaeubl]

I think tracking off-heap memory (UnmanagedMemorySupport and VirtualMemoryProvider) is the only approach that is possible realistically. All Java heap allocations (regardless if by the VM or by the application), will be visible in the heap dump anyways, which is more accurate and easier to use.

Regarding the implementation: it is probably best to use an incremental approach with multiple pull requests. The initial prototype should only contain a minimal amount of features (I assume that quite a few places will need to be changed so that we know the allocation reason).

[roberttoyonaga]

Thanks for the feedback @christianhaeubl. Yes, that makes sense to me. I'll first start with getting "summary" mode working with exclusively off-heap memory.

I assume that quite a few places will need to be changed so that we know the allocation reason

Yes that's correct. Because of this I think I'll scope down the first PR to only support a subset of all the categories. Perhaps just "JFR" and a few others to begin with.

Should I add NMT as a org.graalvm.nativeimage.hosted.Feature (similar to JFR/JMX) or should it always be included? Either way, enabling it at runtime would be optional, like it is in regular Java mode.

One other thing to consider is how the NMT data should be exposed. In Java mode, NMT data can be accessed using jcmd, which isn't possible in Native Image. JFR events are another way that exists in Java mode.

Thanks!

[christianhaeubl]

NMT should be behind a new hosted option. If the impact on image size is negligible, we can convert the option to a runtime option rather easily.

Regarding exposure: for the initial PR, it is probably sufficient to just add a shutdown hook that prints the data. You can of course also do the JFR-part right away, if it is simple enough.

Note that you will probably need to do a few cleanups:

• LibC.malloc() may be called directly at the moment (UnmanagedMemorySupport.malloc(...) should be used instead)
• UnmanagedMemorySupport.free(...) may be called on memory that was allocated outside of Java code (e.g., memory that was allocated by C code or libc functions)

[roberttoyonaga]

@christianhaeubl is there a reason that currently UnmanagedMemory, UnmanagedMemorySupport, and LibC are each called directly in SVM code to handle memory? I'd like to clean things up so that only UnmanagedMemory is publicly available whileUnmanagedMemorySupport and LibC are essentially hidden and used implicitly.

[christianhaeubl]

UnmanagedMemory and UnmanagedMemorySupport have different behavior in out of memory situations (depending on what is needed for certain callers), so I think it makes sense to keep both classes:

• UnmanagedMemory throws an OutOfMemoryException
• UnmanagedMemorySupport just returns null

LibC.malloc(), LibC.realloc(), and LibC.calloc() should only be used by UnmanagedMemorySupport (i.e., all existing calls can be changed to use UnmanagedMemorySupport instead).

Feel free to do those cleanups in a separate PR.