Create heapdump on out of memory

[paulrutter]

Introduction
This issue is a followup of #23328. The outcome of that issue was the introduction of the v8.writeHeapSnapshot() API.

Next step would be to introduce a way of handling an out of memory situation in JS context.
When running nodejs processes in a low memory environment, every out of memory that occurs is interesting. To figure out why a process went out of memory, a heapdump can help a lot.

Desired solution
There are several possible solutions which would suffice:

• introduce an event like process.on('fatal_error'), which kicks in for an OoM event. (see Post-mortem out of memory analysis diagnostics#239 (comment)). The question is if it's feasible to execute JS code after the 'fatal_error' occurred?
• add a CLI flag which enables automatic heapdumps on OoM. This might be more feasible, as it doesn't need to be a JS API and thus can be fixed in native code instead of JS. Downside is that it's not flexible.
• ...

Alternatives
At the moment, we use our own module. This uses native code to hook into the SetOOMErrorHandler handler of V8.
This works although it's not very elegant.

[mmarchini]

I'm not very keen to have this in Node.js core at the moment because it usually takes a lot of memory to take a heap snapshot. If the system is already on a V8 OOM situation, trying to take a heap snapshot can lead to an operating system OOM (which might kill other important processes).

If we could drastically reduce the memory used to take the heap snapshot, I would be happy to see this feature introduced in core.

[joyeecheung]

introduce an event like process.on('fatal_error')

It is possible to execute JS at that point, but in general I don't think it's a good idea to open this opportunity up, as when there is a fatal error we need to be very careful about what we execute (it's similar to signal handlers in some way).

add a CLI flag which enables automatic heapdumps on OoM. This might be more feasible

This looks more promising (or just provide heap snapshots as one of the actions that can be specified to be done when a fatal error occurs, as we already make it possible to trigger node-report in the fatal error handler), though as @mmarchini points out it's at the users' own risk if they want to do that.

[paulrutter]

I would be happy to go for option 2, to make it another configurable cli option. Memory use of the heapdump is indeed a risk, depending on how you use it.
In my usecase, nodejs processes run with restricted old space size, within Docker. So there is always enough memory to make the heapdump. The Docker memory limit has to be at least twice the amount of the nodejs process, to be safe.

[mhdawson]

I'd agree that a CLI flag makes sense, we should look at the existing option for generating a node-report and make it consistent with that.

[matvi3nko]

@paulrutter if you make a snapshot you still will not have anything to compare with, because you need at least one more snapshot. I can assume that the answer to the question: "why do you need a heapdump on OOM" is "inspect state of memory". For this case you can use flag --abort-on-uncaught-exception and create a coredump (stack trace + heapdump) on "process abort". And then you can explore the memory with llnode. It may take a little longer, but it definitely works.
Thanks.

[paulrutter]

@matvi3nko Comparison is only needed when you suspect a memory leak, but this is not always the reason of an out of memory situation. More often, the code being executed is not memory efficient (for example: reading a whole file in memory at once instead of streaming). A single heapdump would show such issues, without the need for comparison.

I tried llnode in the past and found it not very user friendly to use. Of course this is more of an experience issue at my end, but still i think it would be beneficial to other Node.js users to have a more entry-level heapdump generation process in place.
Compare the use of Chrome DevTools to analyze a heapdump to llnode; it's on a whole different level.

[paulrutter]

With the latest Node.js 12.11.1, the node-oom-heapdump module fails with the following message:

<--- JS stacktrace --->
Cannot get stack trace in GC.
Generating Heapdump to 'C:\git\node-oom-heapdump\tests\my_heapdump.heapsnapshot' now...


#
# Fatal error in , line 0
# unreachable code
#
#
#
#FailureMessage Object: 0000006BE0FF7890

The API's for creating a heapdump do not have seem to be changed. It seems that calling "createHeapSnapshot" no longer works in the context that it did before.
But maybe i should ask the v8 team for help on this issue.

Is there any progress made on adding the functionality to Node.js core?

[mmarchini]

Does it work on v12.10?

[paulrutter]

No, it doesn't. Same behavior.

[mmarchini]

What about 12.0.0, 12.3.0 and 12.5.0 (those are the V8 bumps during Node.js v12)? If the issue is on V8 it's good to narrow down which version it started.

--

Also, you should be able to get a core dump if this is throwing a Fatal error. This will allow you to print the native call stack, which should help finding the issue.

To generate a core dump, run:

ulimit -c unlimited
node your-code.js

And then open it with gdb or lldb to get the stack trace:

gdb core  # lldb /cores/core.PID  if your're on OS X
(gdb) bt

Post the core dump stack trace here, should help narrow down the issue.

EDIT: Don't post the core dump, it's a bad idea 😅