Constant disk write from s6-supervise

[ioqy]

Checklist

• Have you pulled and found the error with jc21/nginx-proxy-manager:latest docker image?
  • Yes
• Are you sure you're not using someone else's docker image?
  • Yes
• Have you searched for similar issues (both open and closed)?
  • Yes

Describe the bug
The s6-supervise frontend process is constantly writing to disk at a rate of about 16 kilobyte/s. This amounts to roughly 1.3GB of writes per day.

I also noticed that the backups of my server grew larger in the last weeks. The backups are deduplicated which is why I noticed that the size of the backups grew larger.

I have been using NPM only for a couple of weeks so I can't say if the behaviour of s6-supervise is new. But so far I couldn't find anything else that would explain my larger backups.

Also tried a new NPM container without any configuration at all (no ports, environment and volumes in the docker-compose.yml) with the same result.

I tested it first with 2.9.21 and then upgraded to 2.9.22 to see if the new version makes a difference, but it was the same in both versions.

Edit: Going back to 2.9.20 removed these disk writes completely.

The same issue with s6-supervise but a different container is also described here: linuxserver/docker-ddclient#57

Nginx Proxy Manager Version
v2.9.22

To Reproduce
Steps to reproduce the behavior:

1. Start the docker container
2. Watch the DISK WRITE column of the s6-supervise frontend process with iotop --process --only --accumulate

Expected behavior
A reasonable amount of disk writes.

Screenshots
This is what iotop looks like after about 15 minutes:

Operating System
Operating System: Debian GNU/Linux 11 (bullseye)
Kernel: Linux 5.10.0-21-amd64
Architecture: x86-64
Docker: 23.0.1, build a5ee5b1

Additional context

Docker compose file:

version: "3"

services:
  nginx-proxy-manager:
    image: jc21/nginx-proxy-manager:2.9.22
    container_name: nginx-proxy-manager
    ports:
      - 80:80
      - 443:443
      - 81:81
    environment:
      DISABLE_IPV6: true
    volumes:
      - /srv/docker-services/nginx-proxy-manager/data:/data
      - /srv/docker-services/nginx-proxy-manager/letsencrypt:/etc/letsencrypt
    healthcheck:
      disable: true
    restart: unless-stopped
    networks:
      services-interconnect: {}

networks:
  services-interconnect:
    external: true

Log of the docker container:

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service prepare: starting
❯ Checking folder structure ...
Changing ownership of /data/logs to 0:0
Disabling IPV6 in hosts
❯ Disabling IPV6 in hosts: /etc/nginx/conf.d
  ❯ /etc/nginx/conf.d/default.conf
  ❯ /etc/nginx/conf.d/include/block-exploits.conf
  ❯ /etc/nginx/conf.d/include/ssl-ciphers.conf
  ❯ /etc/nginx/conf.d/include/assets.conf
  ❯ /etc/nginx/conf.d/include/proxy.conf
  ❯ /etc/nginx/conf.d/include/force-ssl.conf
  ❯ /etc/nginx/conf.d/include/letsencrypt-acme-challenge.conf
  ❯ /etc/nginx/conf.d/include/ip_ranges.conf
  ❯ /etc/nginx/conf.d/include/resolvers.conf
  ❯ /etc/nginx/conf.d/production.conf
Disabling IPV6 in hosts
❯ Disabling IPV6 in hosts: /data/nginx
  ❯ /data/nginx/default_host/site.conf
  ❯ /data/nginx/proxy_host/2.conf
  ❯ /data/nginx/proxy_host/7.conf
  ❯ /data/nginx/proxy_host/13.conf
  ❯ /data/nginx/proxy_host/12.conf
  ❯ /data/nginx/proxy_host/1.conf
  ❯ /data/nginx/proxy_host/4.conf
  ❯ /data/nginx/proxy_host/8.conf
  ❯ /data/nginx/proxy_host/3.conf
  ❯ /data/nginx/stream/2.conf
❯ Secrets-init ...

-------------------------------------
 _   _ ____  __  __
| \ | |  _ \|  \/  |
|  \| | |_) | |\/| |
| |\  |  __/| |  | |
|_| \_|_|   |_|  |_|
-------------------------------------

s6-rc: info: service prepare successfully started
s6-rc: info: service nginx: starting
s6-rc: info: service frontend: starting
s6-rc: info: service backend: starting
s6-rc: info: service backend successfully started
❯ Starting backend ...
s6-rc: info: service frontend successfully started
s6-rc: info: service nginx successfully started
❯ Starting nginx ...
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
[3/24/2023] [8:29:29 AM] [Global   ] › ℹ  info      No valid environment variables for database provided, using default SQLite file '/data/database.sqlite'
[3/24/2023] [8:29:29 AM] [Global   ] › ℹ  info      Generating SQLite knex configuration
[3/24/2023] [8:29:29 AM] [Global   ] › ⬤  debug     Wrote db configuration to config file: ./config/production.json
[3/24/2023] [8:29:30 AM] [Migrate  ] › ℹ  info      Current database version: none
[3/24/2023] [8:29:30 AM] [Setup    ] › ℹ  info      Creating a new JWT key pair...
[3/24/2023] [8:29:35 AM] [Setup    ] › ℹ  info      Wrote JWT key pair to config file: /app/config/production.json
[3/24/2023] [8:29:36 AM] [Setup    ] › ℹ  info      Added Certbot plugins certbot-dns-desec~=1.2.1
[3/24/2023] [8:29:36 AM] [Setup    ] › ℹ  info      Logrotate Timer initialized
[3/24/2023] [8:29:36 AM] [Setup    ] › ℹ  info      Logrotate completed.
[3/24/2023] [8:29:36 AM] [IP Ranges] › ℹ  info      Fetching IP Ranges from online services...
[3/24/2023] [8:29:36 AM] [IP Ranges] › ℹ  info      Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
[3/24/2023] [8:29:37 AM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v4
[3/24/2023] [8:29:37 AM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v6
[3/24/2023] [8:29:37 AM] [SSL      ] › ℹ  info      Let's Encrypt Renewal Timer initialized
[3/24/2023] [8:29:37 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
[3/24/2023] [8:29:37 AM] [IP Ranges] › ℹ  info      IP Ranges Renewal Timer initialized
[3/24/2023] [8:29:37 AM] [Global   ] › ℹ  info      Backend PID 108 listening on port 3000 ...
[3/24/2023] [8:29:37 AM] [Nginx    ] › ℹ  info      Reloading Nginx
[3/24/2023] [8:29:37 AM] [SSL      ] › ℹ  info      Renew Complete
[3/24/2023] [8:53:26 AM] [Express  ] › ⚠  warning   invalid signature
[3/24/2023] [9:29:37 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
[3/24/2023] [9:29:37 AM] [Nginx    ] › ℹ  info      Reloading Nginx
[3/24/2023] [9:29:37 AM] [SSL      ] › ℹ  info      Renew Complete

[blaine07]

No idea if this helps any or not, as I am unsure how to provide more information, but you can see the "spike" and consistently "more" disk throughput on my NPM VM since the upgrade yesterday evening. Upgrading to .22 was my only "change."

Edit: showing the last 7 days just for visibility sake of "what it was."

[Mainfrezzer]

The issue exists in 2.9.21 and 2.9.22 for me, 2.9.20 is fine in that regard.

[ioqy]

Thanks for the tip. If I would have looked at the release notes, I could have seen that the s6-overlay was changed in 2.9.21.

I just went back to 2.9.20 and this eliminated these disk writes completely.

[AnSch1510]

I have the same issue. Noticed that my qnap made a constant disk noise and the disk activity lights show activity all the time. As soon as I stop the NPM the disk activity is gone. The Ressource Monitor shows 6 write operations per second continuously as soon as NPM is running. Have rolled back to 2.9.20 and now there is no disk activity anymore.

[4lexf]

The s6-supervise frontend process is constantly writing to disk at a rate of about 16 kilobyte/s. This amounts to roughly 1.3GB of writes per day.

Hi !
Every second supervise executes /etc/s6-overlay/s6-rc.d/frontend/run, try adding " sleep 1d" before "exit 0" in this file.
Maybe code is missing.
...
s6-setuidgid npmuser yarn install
exec s6-setuidgid npmuser yarn watch
else
sleep 1d
exit 0
fi

[Mainfrezzer]

The s6-supervise frontend process is constantly writing to disk at a rate of about 16 kilobyte/s. This amounts to roughly 1.3GB of writes per day.

Hi ! Every second supervise executes /etc/s6-overlay/s6-rc.d/frontend/run, try adding " sleep 1d" before "exit 0" in this file. Maybe code is missing. ... s6-setuidgid npmuser yarn install exec s6-setuidgid npmuser yarn watch else sleep 1d exit 0 fi

That does seem to be a bandaid.
The constant writes of the following files has stopped.

[jslegers73]

Problem is also there with version v2.10.2. I have also used the "bandaid" and it works. Thanks.

[mosaicws]

I'm having same issue running in Docker on Raspberry Pi 4 using usb boot. external ssd activity light constantly flashing explained as described. Hope this gets fixed properly.

How exactly to apply the "bandaid"? Log into the container itself and apply changes?

I can't find the "s6-overlay" folder...

Can this be persisted?

[jslegers73]

Hi yes you need to go into the container.

My steps

docker exec -it Nginx-Proxy-Manager /bin/bash
apt update
apt install nano
cd /etc/s6-overlay/s6-rc.d/frontend/
nano run
exit

Restart the container.

[mosaicws]

Hi yes you need to go into the container.

My steps

docker exec -it Nginx-Proxy-Manager /bin/bash apt update apt install nano cd /etc/s6-overlay/s6-rc.d/frontend/ nano run exit

Restart the container.

Ah ok thanks - I downgraded to 2.9.20 and that folder isn't in that version so I couldn't find it but it's in the latest so all good now.

[jolteondude]

I tried @jslegers73's solution it worked for me, thank you for sharing!
You can also use docker cp in case you don't want to install nano inside the container, here are the steps:

1. sudo docker cp npm_container:/etc/s6-overlay/s6-rc.d/frontend/run . to copy the file to your host machine
2. sudo nano run and add the line "sleep 1d" above "exit 0", it will look like this below

#!/command/with-contenv bash
# shellcheck shell=bash

set -e

# This service is DEVELOPMENT only.

if [ "$DEVELOPMENT" = 'true' ]; then
        . /bin/common.sh
        cd /app/frontend || exit 1
        HOME=/tmp/npmuserhome
        export HOME
        mkdir -p /app/frontend/dist
        chown -R "$PUID:$PGID" /app/frontend/dist

        log_info 'Starting frontend ...'
        s6-setuidgid npmuser yarn install
        exec s6-setuidgid npmuser yarn watch
else
        sleep 1d
        exit 0
fi

3. sudo docker cp run npm_container:/etc/s6-overlay/s6-rc.d/frontend/run to send it back to the container
4. sudo docker restart npm_container to apply it

[glassman81]

The original temp fix was actually suggested by @4lexf, but this simple sed command should also work from within the container:

sed -i '/else/a\\tsleep 1d' /etc/s6-overlay/s6-rc.d/frontend/run

You can then verify it worked by displaying the contents of the file:

cat /etc/s6-overlay/s6-rc.d/frontend/run

Congrats. You're done.

---

If you want something more permanent for when the container is recreated for any reason, such as a watchtower update, etc., you can make a really simple script like below:

#!/bin/bash
if ! grep -qZF "sleep 1d" /etc/s6-overlay/s6-rc.d/frontend/run; then
   sed -i '/else/a\\tsleep 1d' /etc/s6-overlay/s6-rc.d/frontend/run
fi

You would then save the script in a local location that hosts the docker installation, and map it to the inside of the container as a volume. That specific inside location is /etc/cont-init.d. You'll want to name this local script something like 99-whateveryouwant. When creating the container, you just add the additional volume like so:

-v /your/local/location/99-whateveryouwant:/etc/cont-init.d/99-whateveryouwant

Admittedly, this is a bit overkill since most people won't recreate their container by the time a fix is available (manually or otherwise), but for those that want to set it and forget it, this works.

[misinformed329]

Thank you @4lexf and @glassman81. This seems easy enough to do with that sed command. Does anyone know if this has been fixed in v2.10.3?

[Mainfrezzer]

@misinformed329 its still requires manual fixing in 2.10.3 (alongside the Certbot renewal issue)