JVM Profiling

[bruno-garcia]

Description

Support Sentry profiling running in the JVM

### Milestones
- [ ] Profile Java code in the JVM for single hub mode (desktop and CLI type of app)
- [ ] Profile per-thread/request to support Spring Boot and other web frameworks
- [ ] Support profiling for Spring Boot

The biggest opportunity here is supporting Spring Boot specifically. But generically supporting JVM for both per-thread or per-request (webflux/async) will be the complete deliverable.

Assuming here that if we get Java it won't matter the actual language that compiled to bytecode. It would be valuable to support Kotlin, Scala, Clojure, Groovy (in this order, if it mattered).

Related profiling issues and PRs:

.NET: getsentry/sentry-dotnet#1955 and getsentry/sentry-dotnet#2206
iOS profiler: * getsentry/sentry-cocoa#1652
Android: #1897 and #1973
NodeJS: https://github.com/getsentry/profiling-node
Python: getsentry/sentry-python#1481
Ruby: getsentry/sentry-ruby#2013

[vaind]

I've done a bit of research into this and the closest to what we have in other languages and also most useful seems to be the async-profiler which combines AsyncGetCallTrace with perf_events (https://youtu.be/WnRngFMBe7w). It has also been adapted by DD for their use (https://github.com/DataDog/java-profiler, which is a derivative work based on async-profiler)

• https://github.com/async-profiler/async-profiler/wiki
• https://krzysztofslusarski.github.io/2022/12/12/async-manual.html
• https://github.com/openjdk/jmh/blob/master/jmh-core/src/main/java/org/openjdk/jmh/profile/AsyncProfiler.java
• https://github.com/elastic/apm-agent-java/blob/b03881fb01e9e8cabdbdba55ac7b782970704fd7/apm-agent-plugins/apm-profiling-plugin/src/main/java/co/elastic/apm/agent/profiler/SamplingProfiler.java

[vaind]

I've experimented a bit with async-profiler and here are some notes (besides the updated links in the previous comment):

Findings

• Only officially available on Linux and macOS. There is supposed to be an IntelliJ managed version for Windows, but I haven't looked into that yet.
• It is implemented as a native library (.so) and is usually meant as a standalone profiler run from CLI.
• It does, however, have a rudimentary Java API that basically forwards calls to the underlying native profiler via the execute command, for example: profiler.execute("start,jfr,event=cpu,interval=" + profilingRateNs + ",file=test.jfr")
• the seemingly only useful output type is JFR, i.e. java flight recorder format - we should be able to transform that if we wanted to go that route)
• it only supports running a single profiler at a time - this is a problem for Spring Boot - not sure how we could do continuous profiling with thread-based data splitting or single-thread profiling in a multi-threaded environment - there's a thread filter, but I don't see how we could use this for Spring Boot because we would need to constantly add/remove threads but in the end, we wouldn't have the profile until we stopped. With the profiler being a singleton, we can't launch multiple profiler instances and have each one handle a different thread.
• as is the profiler outputs a single file for a single profiling session, we can process the file after the profiling is stopped

Possible solution

Currently, the only possible avenue (I can see) to make this profiler work for our use-cases is by altering the c++ source code to:

• allow running continuously or have a thread-safe stop+resume functionality
• produce partial, Sentry-specific profiles
• support producing profiles for individual threads (Spring Boot)

Problems

Within my limited experiments, I have encountered a SIGBUS after enabling the profiler (happens almost every time) on Java 17.0.7

While this specific crash is caused by a JVM bug, it seems it's not uncommon and there's a lot of crashes reported. I haven't evaluated those yet to see whether async-profiler is to blame or other bugs in JVM, e.g. as the first one...

[apangin]

By a chance, I came across this research from a linked issue.
As a general comment - you could raise a relevant discussion in the async-profiler project or request a consultation; this might have saved you tons of time and lead to the right direction.

Only officially available on Linux and macOS. There is supposed to be an IntelliJ managed version for Windows, but I haven't looked into that yet.

There is no open source version of async-profiler for Windows. Your best bet on Windows is probably JDK Flight Recorder. It comes with a number of flaws, but given the limitations of the platform, it is hard to achieve the same level of Java profiling quality as on Linux or macOS anyway.

It is implemented as a native library (.so) and is usually meant as a standalone profiler run from CLI.

There are 3 main ways to run async-profiler:

1. Standalone CLI utility asprof that can attach to a Java process at runtime.
2. Agent library that is loaded at JVM startup.
3. JNI library loaded at runtime through Java API.

All are equally supported, fully functional and widely used.

It does, however, have a rudimentary Java API

Minimalistic does not mean rudimentary. Async-profiler's Java API is quite mature.

the seemingly only useful output type is JFR

Depends on the definition of "useful". But it's true that JFR is the most comprehensive, flexible and suitable for automated parsing format. It's also de facto standard for Java profiling: there is a standard API for processing JFR recordings, many tools support JFR files: JMC, VisualVM, IntelliJ IDEA etc. Async-profiler has a built-in JFR reader and a number of converters from JFR to other formats.

not sure how we could do continuous profiling with thread-based data splitting or single-thread profiling in a multi-threaded environment

Async-profiler records thread information in JFR recordings by default. For other output formats it can be enabled with -t | threads option. In the wall clock mode, threads can be also included/excluded from profiling dynamically using Java API. I'm not aware of issues with extracting per-thread profiles. If you have one, please contact us through the respective support channels.

we wouldn't have the profile until we stopped

dump command dumps the collected data to a file without stopping profiler. With JFR output, dump forces completion of the current JFR chunk and starts a new one.

we can't launch multiple profiler instances and have each one handle a different thread

Why would this be needed?

as is the profiler outputs a single file for a single profiling session, we can process the file after the profiling is stopped

As mentioned earlier, there is a dump command to get interim profiling results. Furthermore, chunktime/chunksize options will force flushing of JFR chunk every given time interval or number of bytes. Each JFR chunk is self-sufficient and can be parsed independently of other chunks.

allow running continuously or have a thread-safe stop+resume functionality

loop option is intended for continuous profiling.
When using Java API, you may implement custom logic on top of stop/start commands (they are already thread-safe).
See #71 for details.

I have encountered a SIGBUS after enabling the profiler (happens almost every time) on Java 17.0.7

Yes, this is a notorious JVM bug specific to JDK 17.0.7 on macOS/ARM.
Hopefully this will be fixed in the next JDK update, as the fix is straightforward. If not, I'll come up with a workaround in async-profiler, although it won't be as trivial as in JDK itself.

there's a lot of crashes reported

If I search for "crash" in the JDK bugtracker, I'll find 19K issues. Not sure what to conclude from this fact though, except that JDK is quite popular, and being a complex piece of software it sometimes works not as expected. If the project is alive and supported, the issues are continuously discovered and addressed.

[vaind]

Thanks @apangin for your valuable input and for filling in more context. I really appreciate that.

Firstly, to clarify parts of my comment which may have been misunderstood: all the info and assumptions I've written down were in the context of Sentry SDK JVM profiling, not meant as general async-profiler summary. For example, when I was saying JFR is the only useful output, that was meant as the only output supported by async-profiler that is useful for our use-case in sentry-java for JVM profiling.

Similarly, with my other comments, for example saying that the Java API is rudimentary, this may have come along as offensive but was meant as "basic/limited", i.e. not really a fully-fledged API with methods for individual functionalities of the underlying profiler but rather that it had just a few most used methods and the rest was available through the execute(). Maybe my word choice was not the best since English is my second language. Anyway, I did not intend to debase async-profiler, it is a great product and I'm just beginning to see all the effort behind it.

With regards to per-thread and continuous profiling - it seems I have missed/misunderstood some of the behaviour and this will need further investigation on our end. We'll need to find out whether it can be used to achieve the most important use case at hand, which can be simplified as: "collect sample-profiler sessions for individual transactions, which are actually individual requests served by Spring boot". Assuming requests are served on individual threads, this means collecting short profiles for individual threads. And because the requests are served in parallel, we need to run multiple independent (as in independent start&stop and independent data output) profiler sessions in parallel.

[vaind]

Hey, I've gotten back to this and I think the async-profiler with some adaptations is a viable option. As I've already mentioned above, I still feel the best approach (performance wise) would be to fork/derive from the original source and make it produce profile output tailored specifically to Sentry, without the intermediate JFR conversion. I think we could do this by picking the select sources and compiling the library with Gracle CppCompile so we don't have additional complexity with submodules and a separate fork.

Alternatively, I could try integrating the async-profiler as is, with the default profiler and let it dump output to disk in JFR format periodically and then (read & convert to Sentry profile format in our SDK. This may be slightly easier to implement but will have higher runtime overhead due to disk access & conversion. I'm a bit reluctant with this because I think we may end up with an additional layer that would act as a buffer that would be collecting the data from files and slicing it, basically like another continuous profiler but instead of actually profiling, it would be just reading JFR files and combing them to form an in-memory model for the past 30 seconds, so that we can slice it at will on transaction end (like in some other SDKs)

Any thoughts?

[bruno-garcia]

would be to fork/derive from the original source and make it produce profile output tailored specifically to Sentry, without the intermediate JFR conversion.

Agreed!

I think we could do this by picking the select sources and compiling the library with Gracle CppCompile so we don't have additional complexity with submodules and a separate fork.

Wouldn't this make it harder to keep it up-to-date with the changes upstream? I tend to prefer a submodule on the fork

[vaind]

Wouldn't this make it harder to keep it up-to-date with the changes upstream? I tend to prefer a submodule on the fork

We would likely need just a part of the repo but yeah, a fork is an option too, if you so prefer. I agree it's easier to follow the updates.

[bruno-garcia]

👀 https://github.com/glowroot/glowroot

[bruno-garcia]

More context in this thread:

• ✨ Where is Profiling on the Spring Boot / Java feature roadmap? #2907