[GR-38994] Add support for `-XX:+HeapDumpOnOutOfMemoryError`

[galderz]

Feature request

Following on the addition of heap dumps, support for -XX:+HeapDumpOnOutOfMemoryError should be added. This flag enables the heap dump to be dumped in the event of an out of memory error.

Describe who do you think will benefit the most.
-XX:+HeapDumpOnOutOfMemoryError is a flag that is commonly used in Java programs. Now that heap dumps can be retrieved on demand, Java developers would expect this flag to be available. Heap dumps are needed the most when the JVM is running out of memory.

Describe alternatives you've considered.
The alternative would be for a user to somehow be alerted when an OOME is reported and for the user to request a heap dump manually or via some automation.

I'd like to contribute this feature.

[fniephaus]

I assume this is about GraalVM Native Image, not the JVM mode, right?

[thomaswue]

This is a great suggestion. One prerequisite for making this work is to modify the heap dumping code to avoid allocations - or at least stay within some predefined allocation bounds reserved for this feature.

[fniephaus]

I believe @thomaswue is referring to #4637.

[galderz]

Indeed for this to work the heap dump generation has to be virtually allocation free, otherwise you might get an OOME again when trying to dump it. I'll give this a go once #4637 has been resolved.

[jerboaa]

Apparently @christianhaeubl is working on heap dump refactoring. So this needs a bit of coordination @galderz

[TharmiganK]

@christianhaeubl any update on supporting this feature?

[christianhaeubl]

@galderz any update regarding your plans on supporting this feature?

[galderz]

@christianhaeubl I'll have a look now that allocation-free heap dump is in place.