native image build should report RSS size of build process #3879
Description
Feature request
Add resident set size memory information to the native image build process for a more accurate picture of consumed memory.
Is your feature request related to a problem? Please describe.
We frequently analyze the native image build process for memory consumption. While it shows the current heap capacity (via Runtime.getRuntime().totalMemory()), but that's not the total picture as it only showing (some) heap information. There are other things consuming memory. Native GC structures, Class metadata, JIT compiler code caches, etc. This is particularly relevant on cloud systems where the build process might run with a memory limit and the native-image build process gets seemingly killed too early by the Linux OOM killer. The memory reporting of the native-image build process seemingly shows that there is still some room left. That can leave users confused.
Describe the solution you'd like.
Include resident set size (RSS) of the processes' virtual memory in addition to heap consumption for the native image build process. Example:
[spring-to-quarkus-todo-0.0.1-SNAPSHOT-runner:47] classlist: 15,819.63 ms, 1.19 GB 0.71 GB (VmRSS)
[spring-to-quarkus-todo-0.0.1-SNAPSHOT-runner:47] (cap): 691.90 ms, 1.19 GB 0.73 GB (VmRSS)
[spring-to-quarkus-todo-0.0.1-SNAPSHOT-runner:47] setup: 2,464.19 ms, 1.19 GB 0.74 GB (VmRSS)
The bundle named: messages, has not been found. If the bundle is part of a module, verify the bundle name is a fully qualified class name. Otherwise verify the bundle path is accessible in the classpath.
17:07:37,727 INFO [org.hib.Version] HHH000412: Hibernate ORM core version 5.5.7.Final
17:07:37,812 INFO [org.hib.ann.com.Version] HCANN000001: Hibernate Commons Annotations {5.1.2.Final}
17:07:38,044 INFO [org.hib.dia.Dialect] HHH000400: Using dialect: io.quarkus.hibernate.orm.runtime.dialect.QuarkusPostgreSQL10Dialect
17:09:02,211 INFO [org.jbo.threads] JBoss Threads version 3.4.2.Final
[spring-to-quarkus-todo-0.0.1-SNAPSHOT-runner:47] (clinit): 1,169.83 ms, 3.02 GB 3.49 GB (VmRSS)
[spring-to-quarkus-todo-0.0.1-SNAPSHOT-runner:47] (typeflow): 118,958.38 ms, 3.02 GB 3.49 GB (VmRSS)
[spring-to-quarkus-todo-0.0.1-SNAPSHOT-runner:47] (objects): 91,161.64 ms, 3.02 GB 3.49 GB (VmRSS)
[spring-to-quarkus-todo-0.0.1-SNAPSHOT-runner:47] (features): 1,574.82 ms, 3.02 GB 3.49 GB (VmRSS)
[spring-to-quarkus-todo-0.0.1-SNAPSHOT-runner:47] analysis: 215,031.84 ms, 3.02 GB 3.49 GB (VmRSS)
[spring-to-quarkus-todo-0.0.1-SNAPSHOT-runner:47] universe: 7,198.35 ms, 3.02 GB 3.53 GB (VmRSS)
[spring-to-quarkus-todo-0.0.1-SNAPSHOT-runner:47] (parse): 23,000.72 ms, 2.84 GB 3.76 GB (VmRSS)
[spring-to-quarkus-todo-0.0.1-SNAPSHOT-runner:47] (inline): 21,371.34 ms, 2.83 GB 3.70 GB (VmRSS)
[spring-to-quarkus-todo-0.0.1-SNAPSHOT-runner:47] (compile): 214,593.69 ms, 3.04 GB 3.74 GB (VmRSS)
[spring-to-quarkus-todo-0.0.1-SNAPSHOT-runner:47] compile: 261,936.51 ms, 3.04 GB 3.74 GB (VmRSS)
[spring-to-quarkus-todo-0.0.1-SNAPSHOT-runner:47] image: 9,250.12 ms, 2.98 GB 3.78 GB (VmRSS)
[spring-to-quarkus-todo-0.0.1-SNAPSHOT-runner:47] write: 1,459.14 ms, 2.98 GB 3.77 GB (VmRSS)
[spring-to-quarkus-todo-0.0.1-SNAPSHOT-runner:47] [total]: 513,565.72 ms, 2.98 GB 3.77 GB (VmRSS)
# Printing build artifacts to: /project/spring-to-quarkus-todo-0.0.1-SNAPSHOT-runner.build_artifacts.txt
Describe who do you think will benefit the most.
GraalVM users, GraalVM contributors, developers of libraries and frameworks which depend on GraalVM.
Describe alternatives you've considered.
I've done some experiments to hook this onto the existing native-image output. For Linux this can be done via:
native-image <native-image-args> | while read line; do echo -n "$line"; pid=$(echo "$line" | grep '^\[' | awk '{ print $1 }' | sed 's/\[//g' | sed 's/\]//g' | cut -d':' -f2); if [ "${pid}_" != "_" ]; then grep VmRSS /proc/$pid/status | awk '{ print $2 }' | awk '{ val=$1 / 1024; val=val/1024; printf "%5.2f GB (VmRSS)\n", val }'; else echo; fi; done
but this isn't easy to put into all possible build pipelines. A GraalVM patch seems the better approach.
Express whether you'd like to help contributing this feature
Feel free to assign this feature request to myself. I'll get an initial version (for Linux) implemented. If relevant parties can show me the corresponding Windows/Mac APIs I'd be happy to incorporate.