Container upgrade unexpectedly consumes storage space

[rical]

Current Behavior

Upgrading containers can balloon and fill the filesystem.

Symptom

/mnt/var grows with the size of the container root filesystem each upgrade. Eventually resulting in something simiular to:
pod-roadm[7731]: Error: writing blob: adding layer with blob "sha256:...": processing tar file(write /usr/lib/lib.so: no space left on device): exit status 1

Root cause

Multiple copies of the container filesystem are present in /mnt/var/lib/containers/storage/overlay.

/mnt/var/lib/containers/storage/overlay:
drwx------    6 root     root          4096 Aug 11 13:26 25cdaf1d0d2592c1a79295a449d115d3dcffd17fb98e65427f25495e787a4aca
drwx------    6 root     root          4096 Nov  8  2024 26b1ceb4f1abcd435cf4b9faed92f4f31e7d353e06f7557fd9c2887b44fb6dbc
drwx------    6 root     root          4096 Aug  7 09:58 4b9f5a1c18f792695edf78cd4a45a8e4ac7d17631b2ff7d8ecb8a0e4dede91ff
drwx------    6 root     root          4096 Aug  7 11:40 6025cfee44e10b20a68bcf37586e24708db4ec2baa4472f5ddd6994445cbbad7
drwx------    6 root     root          4096 Aug 11 13:37 71ce1071297f288048720bf331b17f0f32eb896d32c88aa97c4ec4985e050e5f
drwx------    5 root     root          4096 Aug 11 13:37 9eaf494b6148db8566450102a8bb58ca114fba3aafda711cd8537b6a2fb88056

These directories represent new layer IDs created by Podman for each container. Ideally, unused layers should be automatically pruned to free up disk space.

A likely fix is to run podman container prune to remove unused containers and associated layers.

Proposed Fix

Infix should handle this automatically by pruning old, dangling containers. We will work on implementing this behavior to prevent recurrence.

Expected Behavior

Old stale Podman storage layers should be pruned, ensuring /mnt/var doesn't balloon and fill the filesystem.

Steps To Reproduce

Install different versions of the same container (to be investigated).

Additional information

No response