Apt cache issues following `ubuntu-latest` update

[andrewmogan]

Over this past weekend, we had several nightly build failures with the message sudo: cvmfs_config: command not found in the cvmfs_config setup step of this action. After finding #33, I found that manually deleting our cvmfs apt cache solved the issue. It would be nice if users didn't have to worry about this happening out of the blue. In the aforementioned issue, @vvolkl states I think it should also be possible to retry without the cache in case of errors. Can this be revisited? Or maybe a config option to delete caches with a key containing cvmfs-apt-cache in case of errors?

Link to failed build.

[ocaisa]

Indeed, I have had a few complaints from people who fork our repositories as they are getting lots of emails about broken CI. This is due to their bad caches...but they don't know that.

[ocaisa]

Looking at the mentioned failed build, the problem is

Installing cvmfs
  Get:1 file:/etc/apt/apt-mirrors.txt Mirrorlist [144 B]
  Hit:2 http://azure.archive.ubuntu.com/ubuntu noble InRelease
  Hit:3 http://azure.archive.ubuntu.com/ubuntu noble-updates InRelease
  Hit:4 http://azure.archive.ubuntu.com/ubuntu noble-backports InRelease
  Hit:5 http://azure.archive.ubuntu.com/ubuntu noble-security InRelease
  Hit:6 https://packages.microsoft.com/repos/azure-cli noble InRelease
  Hit:7 https://packages.microsoft.com/ubuntu/24.04/prod noble InRelease
  Ign:8 http://cvmrepo.s3.cern.ch/cvmrepo/apt noble-prod InRelease
  Hit:9 http://cvmrepo.s3.cern.ch/cvmrepo/apt noble-prod Release
  Reading package lists...
  Reading package lists...
  Building dependency tree...
  Reading state information...
  Some packages could not be installed. This may mean that you have
  requested an impossible situation or if you are using the unstable
  distribution that some required packages have not yet been created
  or been moved out of Incoming.
  The following information may help to resolve the situation:
  
  The following packages have unmet dependencies:
   uuid-dev : Depends: libuuid1 (= 2.39.3-9ubuntu6.2) but 2.39.3-9ubuntu6.3 is to be installed
  E: Unable to correct problems, you have held broken packages.

so the command is erroring, but the error is not caught/handled.

[ocaisa]

It looks like this is happening on

sudo rm -f /var/lib/man-db/auto-update
sudo apt-get -q update

this could be wrapped into

github-action-cvmfs/setup-cvmfs.sh

Lines 6 to 13 in 76db24a

	if [ -n "${APT_CACHE}" ]; then
	echo "::group::Using cache"
	echo "Copying cache from ${APT_CACHE} to system locations..."
	mkdir -p ${APT_CACHE}/archives/ ${APT_CACHE}/lists/
	sudo cp -r ${APT_CACHE}/archives /var/cache/apt
	sudo cp -r ${APT_CACHE}/lists /var/lib/apt
	echo "::endgroup::"
	fi

where you handle the failure by removing the cache again. If the update succeeds, you set an environment variable so it doesn't happen twice.

I might try to make a PR for this...but how to test it?

EDIT: some vibe coding gives me:

sudo rm -f /var/lib/man-db/auto-update
if [ -n "${APT_CACHE}" ]; then
  echo "::group::Using cache"
  echo "Copying cache from ${APT_CACHE} to system locations..."
  mkdir -p ${APT_CACHE}/archives/ ${APT_CACHE}/lists/
  sudo cp -r ${APT_CACHE}/archives /var/cache/apt
  sudo cp -r ${APT_CACHE}/lists /var/lib/apt
  echo "::endgroup::"

  echo "::group::Validating cache (offline)"
  run_update=true
  if sudo unshare -n bash -c "apt-get -q update"; then
    echo "Cache is valid"
    unset run_update
  else
    echo "Cache is invalid, cleaning up"
    sudo rm -rf /var/cache/apt/archives
    sudo rm -rf /var/lib/apt/lists
    echo "::endgroup::"
  fi
fi

if [ -n "${run_update}" ]; then
  echo "::group::No valid cache provided, running apt update"
  sudo apt-get -q update
  echo "::endgroup::"
fi

[wdconinc]

Sorry, I didn't notice this PR before. We should probably set -e for these scripts to fail. I've encountered these errors as well. Tagging @vvolkl in case he hasn't seen this either.

[wdconinc]

It looks like this is happening on

sudo rm -f /var/lib/man-db/auto-update
sudo apt-get -q update

It seems it must be the next line. There is no output Some packages could not be installed. from just these commands.

[wdconinc]

@ocaisa @andrewmogan @vvolkl For you review, #48. Not sure if there's a good cache poisoning CI test we can include...