Skip to content

move utils.sh to scripts/utils.sh + also update script to create tarballs to also cover scripts/ dir #213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Feb 4, 2023
Merged

move utils.sh to scripts/utils.sh + also update script to create tarballs to also cover scripts/ dir #213

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
4e2f6d0
Move bash utils to new scripts dir, add new dir to tarball creation
huebner-m Dec 20, 2022
5749181
Merge branch 'main' into move_bash_utils
boegel Jan 19, 2023
cdc018c
Merge branch 'main' of github.com:EESSI/software-layer into move_bash…
huebner-m Jan 25, 2023
8cac3df
Update of where utils.sh is located when sourcing
huebner-m Jan 25, 2023
3390a2d
Add create_directory_tarballs.sh to github CI
huebner-m Jan 25, 2023
70339ac
Merge branch 'move_bash_utils' of github.com:huebner-m/software-layer…
huebner-m Jan 25, 2023
f513b12
Add CI test for the create_directory_tarballs.sh script
huebner-m Feb 2, 2023
e4702ee
Need to specify the EESSI version to test create_directory_tarballs
huebner-m Feb 2, 2023
c4b9cf5
Fix wildcard usage when building tarball of the scripts dir
huebner-m Feb 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions .github/workflows/tests_scripts.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,9 @@ on:
- build_container.sh
- install_software_layer.sh
- run_in_compat_layer_env.sh
- utils.sh
- scripts/utils.sh
- update_lmod_cache.sh
- create_directory_tarballs.sh
Copy link
Contributor

@boegel boegel Feb 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only makes sure that this workflow is triggered when changes are made to create_directory_tarballs.sh - we're not actually testing create_directory_tarballs.sh directly in this workflow... Could/should we?

Copy link
Contributor Author

@huebner-m huebner-m Feb 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test should work now (it did locally). However, the CI will still fail until this PR is merged because the script downloads a tarball of the main branch to build the final tarballs and the new subdirectory doesn't exist there yet.


pull_request:
branches:
Expand All @@ -16,8 +17,9 @@ on:
- build_container.sh
- install_software_layer.sh
- run_in_compat_layer_env.sh
- utils.sh
- scripts/utils.sh
- update_lmod_cache.sh
- create_directory_tarballs.sh
permissions:
contents: read # to fetch code (actions/checkout)
jobs:
Expand Down Expand Up @@ -49,3 +51,13 @@ jobs:
# force using x86_64/generic, to avoid triggering an installation from scratch
sed -i "s@./EESSI-pilot-install-software.sh@\"export EESSI_SOFTWARE_SUBDIR_OVERRIDE='x86_64/generic'; ./EESSI-pilot-install-software.sh\"@g" install_software_layer.sh
./build_container.sh run /tmp/$USER/EESSI /tmp/install_software_layer.sh

- name: test create_directory_tarballs.sh script
run: |
# scripts need to be copied to /tmp,
# since create_directory_tarballs.sh must be accessible from within build container
cp -a * /tmp/
cd /tmp
./build_container.sh run /tmp/$USER/EESSI /tmp/create_directory_tarballs.sh 2021.12
# check if tarballs have been produced
ls -l *.tar.gz
2 changes: 1 addition & 1 deletion EESSI-pilot-install-software.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ set -- "${POSITIONAL_ARGS[@]}"

TOPDIR=$(dirname $(realpath $0))

source $TOPDIR/utils.sh
source $TOPDIR/scripts/utils.sh

# honor $TMPDIR if it is already defined, use /tmp otherwise
if [ -z $TMPDIR ]; then
Expand Down
2 changes: 1 addition & 1 deletion check_missing_installations.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ fi

LOCAL_TMPDIR=$(mktemp -d)

source $TOPDIR/utils.sh
source $TOPDIR/scripts/utils.sh

source $TOPDIR/configure_easybuild

Expand Down
33 changes: 18 additions & 15 deletions create_init_tarball.sh → create_directory_tarballs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,43 +4,46 @@ SOFTWARE_LAYER_TARBALL_URL=https://github.com/EESSI/software-layer/tarball/main

set -eo pipefail

function echo_green() {
echo -e "\e[32m$1\e[0m"
}

function echo_red() {
echo -e "\e[31m$1\e[0m"
}

function error() {
echo_red "ERROR: $1" >&2
exit 1
}

if [ $# -ne 1 ]; then
echo "Usage: $0 <EESSI version>" >&2
exit 1
fi

version=$1

TOPDIR=$(dirname $(realpath $0))

source $TOPDIR/scripts/utils.sh

# Check if the EESSI version number encoded in the filename
# is valid, i.e. matches the format YYYY.DD
if ! echo "${version}" | egrep -q '^20[0-9][0-9]\.(0[0-9]|1[0-2])$'
then
error "${version} is not a valid EESSI version."
fatal_error "${version} is not a valid EESSI version."
fi

# Create tarball of init directory
tartmp=$(mktemp -t -d init.XXXXX)
mkdir "${tartmp}/${version}"
tarname="eessi-${version}-init-$(date +%s).tar.gz"
curl -Ls ${SOFTWARE_LAYER_TARBALL_URL} | tar xzf - -C "${tartmp}/${version}" --strip-components=1 --wildcards */init/
source "${tartmp}/${version}/init/minimal_eessi_env"
if [ "${EESSI_PILOT_VERSION}" != "${version}" ]
then
error "Specified version ${version} does not match version ${EESSI_PILOT_VERSION} in the init files!"
fatal_error "Specified version ${version} does not match version ${EESSI_PILOT_VERSION} in the init files!"
fi
tar czf "${tarname}" -C "${tartmp}" "${version}"
rm -rf "${tartmp}"

echo_green "Done! Created tarball ${tarname}."

# Create tarball of scripts directory
# Version check has already been performed and would have caused script to exit at this point in case of problems
tartmp=$(mktemp -t -d scripts.XXXXX)
mkdir "${tartmp}/${version}"
tarname="eessi-${version}-scripts-$(date +%s).tar.gz"
curl -Ls ${SOFTWARE_LAYER_TARBALL_URL} | tar xzf - -C "${tartmp}/${version}" --strip-components=1 --wildcards '*/scripts/'
tar czf "${tarname}" -C "${tartmp}" "${version}"
rm -rf "${tartmp}"

echo_green "Done! Created tarball ${tarname}."
0 utils.sh → scripts/utils.sh
View file
Open in desktop
File renamed without changes.
2 changes: 1 addition & 1 deletion update_lmod_cache.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

TOPDIR=$(dirname $(realpath $0))

source $TOPDIR/utils.sh
source $TOPDIR/scripts/utils.sh

if [ $# -ne 2 ]; then
echo "Usage: $0 <path to compat layer directory> <path to software installation prefix>" >&2
Expand Down