parallel/test-child-process-exec-timeout not cleaning up child process on win7

[BethGriggs]

• Version: Seen on 6.9.4
• Platform: Windows (seen on win7)
• Subsystem: Test

The test parallel/test-child-process-exec-timeout is not always cleaning up its' child processes. When I run the test 10 times (tools/test.py parallel/test-process-child-exec-timeout), I get around 4-5 left over child processes.

Occasionally when the processes are left lying around, the parallel test suite 'hangs' on the 1157th test in the parallel test suite. The test does not timeout with the test runner in this instance - it will hang permanently and the test suite will never complete.

/cc @cjihrig @gibfahn

[gibfahn]

cc/ @nodejs/platform-windows

[gibfahn]

Looks like the test was added in #9208.

Based on that also cc/ @santigimeno

[cjihrig]

@BethGriggs does the test pass, even though the child processes are left behind?

[gibfahn]

@cjihrig Yes it does pass, you can run tools/test.py --repeat 10 parallel/test-child-process-exec-timeout and it will finish fine, it's only when you do a full test run on a machine that already has a bunch of test-child-process-exec-timeout processes lying around that one of the tests then hangs indefininitely.

[gdams]

@gibfahn when I was running I was able to recreate the issue running tools/test.py --repeat 100 parallel/test-child-process-exec-timeout with no processes lying around at the start.

[BethGriggs]

The last parallel run I did ended up with three tests hanging - test-child-process-flush-stdio, test-child-process-exec-stdout-stderr-data-string, and test-child-process-exit-code. These tests all hung until I manually killed a few of the test-child-process-exec-timeout child processes.

There seems to be two issues here:

1. parallel/test-child-process-exec-timeout is not always cleaning up its child processes.
2. The test runner is not timing out when these tests are hanging. By default the tests should be timing out after 60seconds, as shown in the screenshot it reached 290seconds (which is when I manually killed some of the leftover child processes).

To reproduce:

1. Run tools\test.py --repeat 10 parallel/test-child-process-exec-timeout to create some leftover processes.
2. Run tools\test.py -J -p tap parallel and the parallel suite will hang about 1 in every 5 times.

[sam-github]

It's too bad exec doesn't allow an ipc channel to be attached to the child.

It is possible that the test can be rewritten so that the child either replaces or augments its internal timeout with a tcp connection to its parent process, and a handler that self-exits when that connection cannot be made, or is lost - this will limit the lifetime of the child to just a bit more than the parent - right now it looks like it won't terminate if its parent terminates.

[cjihrig]

Is it possible to identify which of the three child processes is the problematic one (the test could be modified to pass an additional unique argument to each child)?

Shouldn't the child processes be exiting after their setTimeout() elapses? Could something be going wrong with common.platformTimeout() on Windows? If the timeout is working properly, would adding a forced process.exit() in the child after the console.error() call work?

[sam-github]

This test is pretty subtle. For one thing, node is not the "child" process that is getting killed, the shell is. That means that node never gets signalled, it always times out, and attempts to write its output. I couldn't figure out how it ever passed, even on Linux, so I read child_process. It turns out that the timeout implementation has an undocumented behaviour: before it signals its child (the shell) it destroys the stdout/stderr pipes from the shell. Its because of this pipe destruction that the output is not seen, not because the the child is killed before the timeout. As I understand it, bash on OS X has a unique behaviour where it signals its pgrp on exit causing its child (the tests grandchild) to die, so that's a bit different, but also works.

This reason the test works is so non-obvious that even though it does (on unix), I don't think it works as the author intended.

As for Windows, the description above is confusing. What it sounds like is happening from talking to Jeremiah on my team (EDIT: @Akpotohwo) is that the test never completes, and is eventually killed by the test runner, so the real problem is the test never completing - not that children leak (thats a follow-on effect of the testrunner killing the top-level node process). However, he also says that only the first exec with timeout using SIGTERM is leaking. That is odd, because on Windows SIGTERM and SIGKILL are handled identically (see uv__kill in deps/uv/src/windows), though after the child terminates UV lies and claims it was killed with a signal (it wasn't, Windows has no signals, but the lie makes node more uniform on Windows and Linux).

I don't know why this is, but I'm wondering if there is something about the destruction of the stdout/err pipes that is confusing the child (remember, the child is cmd.exe, not node), or possibly confusing the node grand-child, and preventing its exit on timeout.

[santigimeno]

For one thing, node is not the "child" process that is getting killed, the shell is. That means that node never gets signalled, ...

This is what we observed (on Linux) when looking into test-child-process-exec-kill-throws flakiness. So IIUIC killing a process launched with exec() will never kill the node grandchild process but just the shell, right? If that's the case could it be a good idea documenting that as I think people would expect it do so? Is there a way to propagate the signal to the node grandchild?