Skip to content

Add zstd support for faster tarball creation or extraction in eessi_container.sh #994

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Apr 10, 2025
Merged

Add zstd support for faster tarball creation or extraction in eessi_container.sh #994

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
2d831f6
Add zstd support for faster tarball creation or extraction when --res…
Apr 2, 2025
901dc67
Add SAMtools as a test for the build pipeline speed
Apr 2, 2025
9cad884
Restore original eessi_container.sh
Apr 2, 2025
96bb45f
Fix typo
Apr 2, 2025
6e7b252
Implement zstd support
Apr 2, 2025
1b408d3
restore original container, to show performance
Apr 2, 2025
fe3c227
Revert "restore original container, to show performance"
Apr 2, 2025
8d706de
Remove samtools, add pmt
Apr 2, 2025
17cff91
Restore original container to show original performance
Apr 2, 2025
cffa653
Revert "Restore original container to show original performance"
Apr 2, 2025
1a41987
Remove pmt again, since that was only to demo the speedup of the buil…
Apr 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 30 additions & 4 deletions eessi_container.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -408,6 +408,19 @@ else
echo "Using ${EESSI_HOST_STORAGE} as tmp directory (to resume session add '--resume ${EESSI_HOST_STORAGE}')."
fi

# if ${RESUME} is a file, unpack it into ${EESSI_HOST_STORAGE}
if [[ ! -z ${RESUME} && -f ${RESUME} ]]; then
if [[ "${RESUME}" == *.tgz ]]; then
tar xf ${RESUME} -C ${EESSI_HOST_STORAGE}
# Add support for resuming from zstd-compressed tarballs
elif [[ "${RESUME}" == *.zst && -x "$(command -v zstd)" ]]; then
Copy link
Contributor

@boegel boegel Apr 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not check once if zstd is supported, define a constant, and use that everywhere:

if [[ -x $(command -v zstd) ]]; then
    USE_ZSTD=true
else
    USE_ZSTD=false
fi

Copy link
Collaborator Author

@casparvl casparvl Apr 3, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, you can, but then you have to check if [[ "${USE_ZSTD}" == 'true' ]] (remember, this is bash, not python, there is no boolean in bash.

Since you have to do a test anyway, the current is explicit, equally short, and execution time of command -v is negligible. No point in storing it in an environment variable, then having to remember what the value to check for is (true or True or whatever).

zstd -dc ${RESUME} | tar -xf - -C ${EESSI_HOST_STORAGE}
elif [[ "${RESUME}" == *.zst && ! -x "$(command -v zstd)" ]]; then
fatal_error "Trying to resume from tarball ${RESUME} which was compressed using zstd, but zstd command not found"
fi
echo "Resuming from previous run using temporary storage ${RESUME} unpacked into ${EESSI_HOST_STORAGE}"
fi

# if ${RESUME} is a file (assume a tgz), unpack it into ${EESSI_HOST_STORAGE}
if [[ ! -z ${RESUME} && -f ${RESUME} ]]; then
tar xf ${RESUME} -C ${EESSI_HOST_STORAGE}
Expand Down Expand Up @@ -865,17 +878,30 @@ if [[ ! -z ${SAVE} ]]; then
# ARCH which might have been used internally, eg, when software packages
# were built ... we rather keep the script here "stupid" and leave the handling
# of these aspects to where the script is used
# Compression with zlib may be quite slow. On some systems, the pipeline takes ~20 mins for a 2 min build because of this.
# Check if zstd is present for faster compression and decompression
if [[ -d ${SAVE} ]]; then
# assume SAVE is name of a directory to which tarball shall be written to
# name format: tmp_storage-{TIMESTAMP}.tgz
ts=$(date +%s)
TGZ=${SAVE}/tmp_storage-${ts}.tgz
if [[ -x "$(command -v zstd)" ]]; then
TARBALL=${SAVE}/tmp_storage-${ts}.zst
Copy link
Contributor

@boegel boegel Apr 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is .zst the standard extensions for zstd-compressed tarballs?
I've never seen that anywhere...

Copy link
Collaborator Author

@casparvl casparvl Apr 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it is. https://en.wikipedia.org/wiki/Zstd

I believe if your tar is build with zstd support, you can do -xaf to inflate and it will infer from the extension that it needs to use zstd. But: I guess you could have zstd on your system and not have your tar compiled with zstd support, so my current implementation is more portable.

tar -cf - -C ${EESSI_TMPDIR} . | zstd -T0 > ${TARBALL}
else
TARBALL=${SAVE}/tmp_storage-${ts}.tgz
tar czf ${TARBALL} -C ${EESSI_TMPDIR} .
fi
else
# assume SAVE is the full path to a tarball's name
TGZ=${SAVE}
TARBALL=${SAVE}
# if zstd is present and a .zst extension is asked for, use it
if [[ "${SAVE}" == *.zst && -x "$(command -v zstd)" ]]; then
tar -cf - -C ${EESSI_TMPDIR} . | zstd -T0 > ${TARBALL}
else
tar czf ${TARBALL} -C ${EESSI_TMPDIR}
fi
fi
tar czf ${TGZ} -C ${EESSI_TMPDIR} .
echo "Saved contents of tmp directory '${EESSI_TMPDIR}' to tarball '${TGZ}' (to resume session add '--resume ${TGZ}')"
echo "Saved contents of tmp directory '${EESSI_TMPDIR}' to tarball '${TARBALL}' (to resume session add '--resume ${TARBALL}')"
fi

# TODO clean up tmp by default? only retain if another option provided (--retain-tmp)
Expand Down
2 changes: 1 addition & 1 deletion test_suite.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,7 @@ else
fatal_error "Failed to extract names of tests to run: ${REFRAME_NAME_ARGS}"
exit ${test_selection_exit_code}
fi
# Allow people deploying the bot to overrwide this
# Allow people deploying the bot to override this
if [ -z "$REFRAME_SCALE_TAG" ]; then
REFRAME_SCALE_TAG="--tag 1_node"
fi
Expand Down
Loading