Skip to content

Conversation

@trz42
Copy link
Collaborator

@trz42 trz42 commented Aug 19, 2024

This PR is essentially redoing #550. Original description below. It is also doing a little code polishing replacing TAB with spaces.


Use case for this is that we want to test software additions to a CernVM-FS repository. Typically we would first run eessi_container.sh with --access rw to make the repository writable with an overlayfs. The changes are stored in some overlay-upper directory (under some /tmp dir or in a tarball of that /tmp directory). When we want to test the additions later we can run eessi_container.sh with --resume TMPDIR_OR_TMPTARBALL. However, until now this only worked when using --access rw which would make the repository again writable, which is likely not what is wanted in a test (because in actual use /cvmfs/* mounts are read-only). In fact, the test-suite.sh run after a build job has finished, uses --access ro (read-only), but this access mode did not recognise changes in an overlay-upper directory.

This PR merges an existing overlay-upper (given via --resume ...) on top of a CernVM-FS repository. Because it only uses the lowerdir option of fuse-overlayfs the merged directory is mounted read-only.


Because the mounting of directories under /cvmfs is done late in the eessi_container.sh script, this should also work with the changes introduced in #618, e.g., one should be able to run something like

eessi_container.sh --repository software.eessi.io,access=ro --repository dev.eessi.io,access=rw --storage=/tmp

which reports about a temporary location used such as /tmp/eessi.abc123defg and then

eessi_container.sh --repository software.eessi.io,access=ro --repository dev.eessi.io,access=ro --resume /tmp/eessi.abc123defg

@trz42 trz42 added enhancement New feature or request 2023.06-software.eessi.io 2023.06 version of software.eessi.io labels Aug 19, 2024
@eessi-bot
Copy link

eessi-bot bot commented Aug 19, 2024

Instance eessi-bot-mc-aws is configured to build for:

  • architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
  • repositories: eessi.io-2023.06-compat, eessi-hpc.org-2023.06-software, eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software

@eessi-bot
Copy link

eessi-bot bot commented Aug 19, 2024

Instance eessi-bot-mc-azure is configured to build for:

  • architectures: x86_64/amd/zen4
  • repositories: eessi.io-2023.06-compat, eessi-hpc.org-2023.06-compat, eessi-hpc.org-2023.06-software, eessi.io-2023.06-software

# the ${RESUME} directory or tarball
# to be able to see the contents of the read-write session we have to mount
# the fuse-overlayfs (in read-only mode) on top of the CernVM-FS repository

Copy link
Collaborator

@casparvl casparvl Aug 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we print something to stdout here to notify that we have detected an overlay directory from a previous build, and are mounting it read-only? I think that will help with future debugging if we ever have issues.

EESSI_FUSE_MOUNTS+=("--fusemount" "${EESSI_READONLY_OVERLAY}")
export EESSI_FUSE_MOUNTS
else
# basic "ro" access that doesn't require any fuseoverlay-fs
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe also add an explicit echo here, just so we easily see in which branch we went.

Copy link
Collaborator

@casparvl casparvl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once the two explicit echo's are added to make it more transparent into which branch we went, I'd say this is good to go.

#676 (comment) proves that this enables the test step of running in ro mode with the previous overlay-upper mounted as lowerdir (i.e. read-only). It shows that the patchelf we just built in the build step is available in the test step. At the same time, #677 proves that without this PR (#674 ), the patchelf we just built in the build step is not available in the test step.

In summary: this PR does exactly what it says on the label :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

2023.06-software.eessi.io 2023.06 version of software.eessi.io enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants