oops, unknown child exited with code 0

[derekrprice]

Description

The following code:

No code needed. Just start php-fpm.

Resulted in this output:

2022-03-30T18:30:39.378169050Z + php-fpm
2022-03-30T18:30:39.437828786Z [30-Mar-2022 18:30:39] NOTICE: fpm is running, pid 1
2022-03-30T18:30:39.438452091Z [30-Mar-2022 18:30:39] NOTICE: ready to handle connections
2022-03-30T18:30:52.079271289Z 127.0.0.1 -  30/Mar/2022:18:30:52 +0000 "GET /status" 200
2022-03-30T18:30:59.523826247Z 127.0.0.1 -  30/Mar/2022:18:30:59 +0000 "GET /status" 200
2022-03-30T18:30:59.527060870Z [30-Mar-2022 18:30:59] ALERT: oops, unknown child (91) exited with code 0. Please open a  (https://github.com/php/php-src/issues).
2022-03-30T18:31:04.811549398Z 127.0.0.1 -  30/Mar/2022:18:31:04 +0000 "GET /status" 200
2022-03-30T18:31:04.813610713Z [30-Mar-2022 18:31:04] ALERT: oops, unknown child (104) exited with code 0. Please open a bug report (https://github.com/php/php-src/issues).
2022-03-30T18:31:04.828971625Z [30-Mar-2022 18:31:04] NOTICE: Finishing ...
2022-03-30T18:31:04.831711045Z [30-Mar-2022 18:31:04] NOTICE: exiting, bye-bye!

But I expected it to not exit.

More details:
This was happening with PHP 7.4 and Laravel 8. The docker image that this was running in came up fine under Kubernetes (k8s) 1.19, but began crashing like this with k8s 1.20. I have a good dozen other services running Laravel 6.5 in the same php:7.4-fpm base image with no problems in k8s 1.20. A second service running Laravel 8 on php:7.4-fpm is running into the same problem. I can also still bring up this container in a non-k8s docker environment with no issues.

I have now upgraded my k8s cluster all the way to 1.22.6 and the problem has not gone away. I proceeded to update to the php:8.0.17-fpm docker base image and have updated all my composer-installed packages to their latest minor versions. I am at somewhat of a loss for how to debug. It appears that this must be some sort of interaction specific to Laravel 8 and k8s 1.20+, but the php-fpm error message said to open a ticket here, so I'm hoping that you might be able to provide some insight.

Thanks!

PHP Version

PHP 8.0.17

Operating System

Kubernetes 1.20+ Laravel 8

[derekrprice]

Removed Laravel from the equation and I have this narrowed down to a health-check script that is calling the /status endpoint. So, this DOES appear to be some bad interaction between php-fpm & k8s 1.20.

[derekrprice]

Actually, it's not exactly an incompatibility.  Without changing the container, moving from k8s 1.19 to 1.20 caused php-fpm's /status endpoint to begin intermittently taking longer.  The previous 2s timeout on the healthcheck was hit more frequently and k8s killed the container.  I think the "Oops, unknown child" warnings had to do with k8s killing some health check processes before they completed.

There is still an issue here, I think, since performance was impacted as of k8s 1.20 with no other changes, but I think the issue is more likely with Kubernetes than with PHP FPM.

[cdaguerre]

@derekrprice we are having the same problem. Were you using https://github.com/renatomefi/php-fpm-healthcheck?

[derekrprice]

It was based on that script, yes.

[pimjansen]

Same here! on 1.21 k8s en php 8.1.9. No idea what is causing this

[derekrprice]

I'm not sure I ever figured why this happened in conjunction with the k8s upgrade, or if that was just coincidence, but the failing health checks turned out to be due to an FPM port (9000) accidentally exposed to the internet. A worm would find us within 10 minutes of pod restarts and install a cryptominer which chewed up all the local resources until k8s killed it.

Anyhow, this isn't php-fpm's fault. Port 9000 was never designed to be secure.

[pimjansen]

Yeah well but in my case its not listing to the internet at all. Only receiving traffic via nginx

[derekrprice]

Could something else be causing your health check to fail?

[janosmiko]

In our case, we used the default Kubernetes probe timeout (which is 1 second). If the client sends a bunch of long requests, the php-fpm worker pool was not able to serve the healthcheck request in time on top of the long-running requests and K8s thought the container is dead and restarted it.
I was able to solve this by increasing the liveness probe/readiness probe timeoutSeconds setting.

[jaydrogers]

For those arriving here from Google, I wanted to share an additional solution to this message.

@janosmiko pointed me in a great direction with the health check delays.

I adjusted my health delays to wait a significant amount of time, but I was still getting errors.

Depending on your container structure, it could also be related to your init system.

I'm using S6 Overlay in my container, and I had to enable a special option to not attach to the parent process.

I have a more detailed solution on how I resolved this error in my case here: renatomefi/php-fpm-healthcheck#56 (comment)

[kamazee]

Also for those who google the issue; the reason can be exec call (or alike) on a command with & at the end, for example

<?php
exec('sleep 1 &');

& makes the command go to background which technically means that its parent will be a process with PID=1.
In k8s, a container usually get a dedicated process namespace, and php-fpm usually gets the PID=1, so the sleep from the example becomes a child of php-fpm, hence the ALERT in logs when the process exits.

Also, if you're not sure how to catch what is the "offending" background process, I ran this in the container:

while true; do ps -f --no-headers --ppid=1; done | grep -vF 'php-fpm'

it continuously lists all children of a process with pid=1 and then greps out php-fpm itself. You might need to adjust the grep part to your needs. It will show the command that runs and it should be fairly easy to pinpoint it to the source code.
Of course, just running ps in a loop is suboptimal, so you might want to use more advanced techniques but they require more knowledge, skills, and kernel features to use them.

[bukka]

This was a very helpful discussion that made me realise that this should not really be an alert. I have created an issue #10315 and PR #10319 that fixes it. As I noted there, if anyone thinks that there are potentially more use case than the listed ones, please comment on it.

In addition this also gave me ideas for few improvements in FPM:

• FPM health check  #10314
• FPM support for S6 service startup notifications #10316