Boot nginx from the backend app to fix graceful shutdown in production

[jtgeibel]

In production we aren't gracefully shutting down as expected. nginx
does a quick shutdown when receiving a SIGTERM and once it exits, it
appears that our process is aborted.

Instead of spawning the backend from the shell script, the backend
process is now responsible for spawning the shell script. This allows
the backend to finish its shutdown work even if the script has already
exited.

cat is passed to the nginx buildpack script as our "server", which
will block on STDIN so that nginx will run until receiving
SIGTERM.

r? @pietroalbini

[jtgeibel]

Deployed and tested on staging:

Mar 16 20:17:59 staging-crates-io app/web.1 downloads_counter shard=21 counted_versions=0 counted_downloads=0 pending_downloads=1
Mar 16 20:18:00 staging-crates-io heroku/web.1 Restarting
Mar 16 20:18:00 staging-crates-io heroku/web.1 State changed from up to starting
Mar 16 20:18:01 staging-crates-io app/web.1 downloads_counter shard=22 counted_versions=0 counted_downloads=0 pending_downloads=1
Mar 16 20:18:01 staging-crates-io heroku/web.1 Stopping all processes with SIGTERM
Mar 16 20:18:01 staging-crates-io app/web.1 Starting graceful shutdown
Mar 16 20:18:01 staging-crates-io app/web.1 Persisting remaining downloads counters
Mar 16 20:18:01 staging-crates-io app/web.1 downloads_counter all_shards counted_versions=1 counted_downloads=1 pending_downloads=0
Mar 16 20:18:01 staging-crates-io app/web.1 Server has gracefully shutdown!
Mar 16 20:18:01 staging-crates-io heroku/web.1 Process exited with status 0

[pietroalbini]

Thanks for the PR! Your approach works great, but I'm not sure spawning start-web.sh from the application is the best approach from a code organization structure. I'd prefer if Heroku still calls start-web.sh and the script also launches the server in the background, like this:

#!/bin/bash
set -ue

if [[ -z "${USE_FASTBOOT-}" ]]; then
    unset USE_FASTBOOT
else
    export USE_FASTBOOT
    node --optimize_for_size --max_old_space_size=200 fastboot.js &
fi

# Since this script is launched separately, we tell the nginx
# buildpack (`bin/start-nginx`) that `cat` is our server.
bin/start-nginx cat &
target/release/server &
wait -n

[jtgeibel]

I haven't tried that exact script, but I don't think that will work. I tried a dozen different ways to launch it from the script, and couldn't get any of them to work. Launching server in the foreground, or bringing it the foreground with fg. Trapping SIGTERM in the script. Using bash -c "./script/start-web.sh & ./target/release/server" in the Procfile.

I just couldn't convince bash to work. In particular, wait has an unexpected interaction with traps (and wait -n stops waiting as soon as the first child exits).

When bash is waiting for an asynchronous command via the wait builtin, the reception of a signal for which a trap has been set will cause the wait builtin to return immediately with an exit status greater than 128, immediately after which the trap is executed.

I even had a few scripts that seemed to work fine locally, but just didn't work on Heroku. My guess is that maybe I was running into differences between interactive and non-interactive shells. (For example, I had to use set -m to make sure job control was enabled on Heroku.) My other guess, is that maybe Heroku kills all processes on shutdown as soon as the parent process exits.

As strange as it is to launch nginx from our backened app, it was the only approach I could find that ensured our process lives long enough to clean up.

[pietroalbini]

Ok then, let's get this fixed!

@bors r+

[bors]

📌 Commit 0061c5f has been approved by pietroalbini

[bors]

⌛ Testing commit 0061c5f with merge e5171ce...

[bors]

☀️ Test successful - checks-actions
Approved by: pietroalbini
Pushing e5171ce to master...