Syncing code-base and CVMFS repository with EESSI (January 2024)

[TopRichard]

NESSI and EESSI have diverted, so in the attempt of syncing them back as close as possible, the following PRs have been merged:

NOTE: CUDA/NVIDIA MODIFICATIONS ARE NOT TESTED!!!

• Modifying directory structure + drop pilot terminology #217 --> reverted back, the changes were many and the outcome wasn't as expected
• switch back to using archspec #218
• lots of changes to remove 'pilot' #220
• moved easystack files to new directory structure #221
• subtle changes to make clear this is for NESSI #222
• Changes within the init-files #224
• add most changes (except NVIDIA related ones) to bot/build.sh #226
• Further changes to get closer to EESSI #225
• added symlink for bot/check-result.sh #227
• dummy scripts for job step to test software build #228
• add bot/inspect.sh script #229
• Another step of syncing with EESSI  #231
• Final steps in Syncing with EESSI #232
• Adding NVIDIA-modification-to PR226-bot-build-sync with EESSI #234
• add README.md for easystacks #235
• removed easystack files for 2021.06 and 2021.12 #236
• using pilot.nessi.no in testing eessi_container.sh #237
• add CI for testing licenses #238
• remove pilot from CI #239
• updated CI for testing archdetect #240
• renamed CI file and ported changes from EESSI #241

Test PRs

• {2023.06}[system] EasyBuild V4.9.0 #216
• {2023.06}[system] EasyBuild v4.9.0 (for test purposes only) #219
• {2023.06}[GCCcore/11.3.0] gnuplot v5.4.4 #230 👍
• {2023.06}[GCCcore/12.3.0-CUDA/12.1.1] #233 👍

For now (EESSI Merged PR#454) there exist differences within 74 files as listed below, must carry out resolving those differences carefully:

• .github/workflows/test-software.eessi.io.yml
  • renamed CI file and ported changes from EESSI #241
• .github/workflows/test_eessi_container_script.yml
  • using pilot.nessi.no in testing eessi_container.sh #237
• .github/workflows/test_licenses.yml
  • add CI for testing licenses #238
• .github/workflows/tests_archdetect.yml
  • updated CI for testing archdetect #240
• .github/workflows/tests_scripts.yml
  • remove pilot from CI #239
• EESSI-install-software.sh
  • moved easystack files to new directory structure #221 (directory structure changes)
  • TODO NVIDIA related changes --> addressed in PR#232
• README.md --> differences are because of NESSI vs EESSI
• bot/build.sh
  • add most changes (except NVIDIA related ones) to bot/build.sh #226
  • Adding NVIDIA-modification-to PR226-bot-build-sync with EESSI #234
• bot/check-result.sh
  • added symlink for bot/check-result.sh #227
• bot/check-test.sh
• bot/test.sh
  • dummy scripts for job step to test software build #228 (requires changes of the main job script run by bot instances)
• bot/inspect.sh
  • add bot/inspect.sh script #229
• build_container.sh
  • lots of changes to remove 'pilot' #220 (removed by this PR)
• check_missing_installations.sh --> differences are because of NESSI vs EESSI
• configure_easybuild --> addressed in PR#225, requires rebuild of gnuplot as a test(after the removal of Lua)which we already don't have
• create_directory_tarballs.sh --> addressed in PR#225
• create_lmodrc.py --> addressed in PR#232 requires some naming changes but will carry that on while testing CUDA modifications
• create_tarball.sh --> addressed in PR#225
• easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021a.yml --> moved to easystacks/pilot.nessi.no/2023.06
• easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021b.yml --> moved to easystacks/pilot.nessi.no/2023.06
• easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022a.yml --> moved to easystacks/pilot.nessi.no/2023.06
• easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022b.yml --> moved to easystacks/pilot.nessi.no/2023.06
• easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-system.yml --> moved to easystacks/pilot.nessi.no/2023.06
• easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.0-system.yml --> moved to easystacks/pilot.nessi.no/2023.06
• easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-2022a.yml --> moved to easystacks/pilot.nessi.no/2023.06
• easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.2-2022a.yml --> moved to easystacks/pilot.nessi.no/2023.06
• easystacks/software.eessi.io/2023.06/README.md
  • add README.md for easystacks #235
• easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-001-system.yml --> moved here
• (LATER) easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-010-CUDA.yml --> Will be handled later
• easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2022b.yml --> moved here
• easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2023a.yml --> moved here
• (LATER) easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-001-system.yml --> Will be handled later
• (LATER) easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2022b.yml --> Will be handled later
• (LATER) easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023a.yml --> Will be handled later
• (LATER) easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023b.yml --> Will be handled later
• eb_hooks.py --> addressed in PR#232
• eessi-2021.06.yml
  • removed easystack files for 2021.06 and 2021.12 #236
• eessi-2021.12.yml
  • removed easystack files for 2021.06 and 2021.12 #236
• eessi_container.sh --> addressed in PR#225, also addresses
  some GPU modifications, check the note in the PR
• init/Magic_Castle/bash --> addressed in PR#222
• init/Magic_Castle/eessi_python3 --> addressed in PR#222
• init/README.md --> addressed in PR#222
• init/arch_specs/eessi_arch_arm.spec --> addressed in PR#224
• init/bash --> addressed in PR#224
• init/eessi_archdetect.sh --> addressed in PR#224
• init/eessi_defaults --> addressed in PR#218
  difference in EESSI_CVMFS_REPO_OVERRIDE and using archspec instead of archdetect
• init/eessi_environment_variables --> addressed in PR#224
• init/eessi_software_subdir_for_host.py --> addressed in PR#222
• init/minimal_eessi_env --> addressed in PR#222
• install_scripts.sh --> addressed in PR#232
• licenses/README.md --> addressed in PR#231
• licenses/licenses.json --> addressed in PR#231
• licenses/spdx.py --> addressed in PR#231
• load_easybuild_module.sh --> addressed in PR#232
• scripts/gpu_support/nvidia/install_cuda_host_injections.sh --> addressed in PR#232
• scripts/gpu_support/nvidia/link_nvidia_host_libraries.sh --> addressed in PR#232
• scripts/utils.sh --> addressed in PR#232
• tests/archdetect/aarch64/arm/neoverse-n1/AWS-awslinux-graviton2.output --> addressed in PR#231
• tests/archdetect/aarch64/arm/neoverse-n1/Azure-Ubuntu20-Altra.output --> addressed in PR#231
• tests/archdetect/aarch64/arm/neoverse-v1/AWS-awslinux-graviton3.output --> addressed in PR#231
• tests/archdetect/aarch64/neoverse_n1/AWS-awslinux-graviton2.all.output --> addressed in PR#231
• tests/archdetect/aarch64/neoverse_n1/AWS-awslinux-graviton2.cpuinfo --> addressed in PR#231
• tests/archdetect/aarch64/neoverse_n1/AWS-awslinux-graviton2.output --> addressed in PR#231
• tests/archdetect/aarch64/neoverse_n1/Azure-Ubuntu20-Altra.all.output --> addressed in PR#231
• tests/archdetect/aarch64/neoverse_n1/Azure-Ubuntu20-Altra.cpuinfo --> addressed in PR#231
• tests/archdetect/aarch64/neoverse_n1/Azure-Ubuntu20-Altra.output --> addressed in PR#231
• tests/archdetect/aarch64/neoverse_v1/AWS-awslinux-graviton3.all.output --> addressed in PR#231
• tests/archdetect/aarch64/neoverse_v1/AWS-awslinux-graviton3.cpuinfo --> addressed in PR#231
• tests/archdetect/aarch64/neoverse_v1/AWS-awslinux-graviton3.output --> addressed in PR#231
• tests/archdetect/ppc64le/power9le/unknown-power9le.all.output --> addressed in PR#231
• tests/archdetect/x86_64/amd/zen2/Azure-CentOS7-7V12.all.output --> addressed in PR#231
• tests/archdetect/x86_64/amd/zen3/Azure-CentOS7-7V73X.all.output --> addressed in PR#231
• tests/archdetect/x86_64/intel/haswell/archspec-linux-E5-2680-v3.all.output --> addressed in PR#231
• tests/archdetect/x86_64/intel/skylake_avx512/archspec-linux-6132.all.output --> addressed in PR#231

[trz42]

First complete pass (except for some easystack files) is done. Remaining todos:

• do second pass on current differences

  git diff nessi_2023.06 2023.06-software.eessi.io

  • .github/workflows/test-pilot.nessi.no.yml
    only differences due to repo naming, repo config, supported CPU architectures --> ✅

    diff

    diff --git a/.github/workflows/test-pilot.nessi.no.yml b/.github/workflows/test-software.eessi.io.yml
    similarity index 76%
    rename from .github/workflows/test-pilot.nessi.no.yml
    rename to .github/workflows/test-software.eessi.io.yml
    index 6342d5d..3ac341a 100644
    --- a/.github/workflows/test-pilot.nessi.no.yml
    +++ b/.github/workflows/test-software.eessi.io.yml
    @@ -1,5 +1,5 @@
     # documentation: https://help.github.com/en/articles/workflow-syntax-for-github-actions
    -name: Check for missing software installations in pilot.nessi.no
    +name: Check for missing software installations in software.eessi.io
     on: [push, pull_request, workflow_dispatch]
     permissions:
       contents: read # to fetch code (actions/checkout)
    @@ -12,37 +12,40 @@ jobs:
             EESSI_VERSION:
             - 2023.06
             EESSI_SOFTWARE_SUBDIR_OVERRIDE:
    -#        - aarch64/generic
    +        - aarch64/generic
    +        - aarch64/neoverse_n1
    +        - aarch64/neoverse_v1
             - x86_64/amd/zen2
    -        - x86_64/intel/broadwell
    +        - x86_64/amd/zen3
    +        - x86_64/intel/haswell
             - x86_64/intel/skylake_avx512
             - x86_64/generic
         steps:
             - name: Check out software-layer repository
               uses: actions/checkout@93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 # v3.1.0
    
    -        - name: Mount NESSI CernVM-FS repository
    +        - name: Mount EESSI CernVM-FS pilot repository
               uses: cvmfs-contrib/github-action-cvmfs@d4641d0d591c9a5c3be23835ced2fb648b44c04b # v3.1
               with:
    -              cvmfs_config_package: https://github.com/NorESSI/filesystem-layer/releases/download/latest/cvmfs-config-nessi_latest_all.deb
    +              cvmfs_config_package: https://github.com/EESSI/filesystem-layer/releases/download/latest/cvmfs-config-eessi_latest_all.deb
                   cvmfs_http_proxy: DIRECT
    -              cvmfs_repositories: pilot.nessi.no
    +              cvmfs_repositories: software.eessi.io
    
             - name: Test check_missing_installations.sh script
               run: |
                   export EESSI_SOFTWARE_SUBDIR_OVERRIDE=${{matrix.EESSI_SOFTWARE_SUBDIR_OVERRIDE}}
    -              source /cvmfs/pilot.nessi.no/versions/${{matrix.EESSI_VERSION}}/init/bash
    +              source /cvmfs/software.eessi.io/versions/${{matrix.EESSI_VERSION}}/init/bash
                   # set $EESSI_CPU_FAMILY to the CPU architecture that corresponds to $EESSI_SOFTWARE_SUBDIR_OVERRIDE (part before the first slash),
                   # to prevent issues with checks in the Easybuild configuration that use this variable
                   export EESSI_CPU_FAMILY=${EESSI_SOFTWARE_SUBDIR_OVERRIDE%%/*}
                   module load EasyBuild
                   which eb
                   eb --version
    -              export EESSI_PREFIX=/cvmfs/pilot.nessi.no/versions/${{matrix.EESSI_VERSION}}
    +              export EESSI_PREFIX=/cvmfs/software.eessi.io/versions/${{matrix.EESSI_VERSION}}
                   export EESSI_OS_TYPE=linux
                   env | grep ^EESSI | sort
    -              echo "just run check_missing_installations.sh (should use easystacks/pilot.nessi.no/${{matrix.EESSI_VERSION}}/eessi-${{matrix.EESSI_VERSION}}-*.yml)"
    -              for easystack_file in $(ls easystacks/pilot.nessi.no/${{matrix.EESSI_VERSION}}/eessi-${{matrix.EESSI_VERSION}}-eb-*.yml); do
    +              echo "just run check_missing_installations.sh (should use easystacks/software.eessi.io/${{matrix.EESSI_VERSION}}/eessi-${{matrix.EESSI_VERSION}}-*.yml)"
    +              for easystack_file in $(ls easystacks/software.eessi.io/${{matrix.EESSI_VERSION}}/eessi-${{matrix.EESSI_VERSION}}-eb-*.yml); do
                       echo "check missing installations for ${easystack_file}..."
                       ./check_missing_installations.sh ${easystack_file}
                       ec=$?
    @@ -52,14 +55,14 @@ jobs:
             - name: Test check_missing_installations.sh with missing package (GCC/8.3.0)
               run: |
                   export EESSI_SOFTWARE_SUBDIR_OVERRIDE=${{matrix.EESSI_SOFTWARE_SUBDIR_OVERRIDE}}
    -              source /cvmfs/pilot.nessi.no/versions/${{matrix.EESSI_VERSION}}/init/bash
    +              source /cvmfs/software.eessi.io/versions/${{matrix.EESSI_VERSION}}/init/bash
                   # set $EESSI_CPU_FAMILY to the CPU architecture that corresponds to $EESSI_SOFTWARE_SUBDIR_OVERRIDE (part before the first slash),
                   # to prevent issues with checks in the Easybuild configuration that use this variable
                   export EESSI_CPU_FAMILY=${EESSI_SOFTWARE_SUBDIR_OVERRIDE%%/*}
                   module load EasyBuild
                   which eb
                   eb --version
    -              export EESSI_PREFIX=/cvmfs/pilot.nessi.no/versions/${{matrix.EESSI_VERSION}}
    +              export EESSI_PREFIX=/cvmfs/software.eessi.io/versions/${{matrix.EESSI_VERSION}}
                   export EESSI_OS_TYPE=linux
                   env | grep ^EESSI | sort
                   # create dummy easystack file with a single entry (something that is not installed in EESSI)

  • .github/workflows/test_eessi_container_script.yml

    diff

    diff --git a/.github/workflows/test_eessi_container_script.yml b/.github/workflows/test_eessi_container_script.yml
    index 3bb67b4..33122e6 100644
    --- a/.github/workflows/test_eessi_container_script.yml
    +++ b/.github/workflows/test_eessi_container_script.yml
    @@ -90,15 +90,15 @@ jobs:
                   elif [[ ${{matrix.SCRIPT_TEST}} == 'readwrite' ]]; then
                     outfile=out_readwrite.txt
                     fn="test_${RANDOM}.txt"
    -                echo "touch /cvmfs/pilot.nessi.no/${fn}" > test_script.sh
    +                echo "touch /cvmfs/software.eessi.io/${fn}" > test_script.sh
                     chmod u+x test_script.sh
                     export SINGULARITY_BIND="$PWD:/test"
                     ./eessi_container.sh --verbose --access rw --mode run /test/test_script.sh > ${outfile}
    
                     tmpdir=$(grep "\-\-resume" ${outfile} | sed "s/.*--resume \([^']*\).*/\1/g")
                     # note: must use '--access rw' again here, since touched file is in overlay upper dir
    -                ./eessi_container.sh --verbose --resume ${tmpdir} --access rw --mode shell <<< "ls -l /cvmfs/pilot.nessi.no/${fn}" > ${outfile}
    -                grep "/cvmfs/pilot.nessi.no/${fn}$" $outfile
    +                ./eessi_container.sh --verbose --resume ${tmpdir} --access rw --mode shell <<< "ls -l /cvmfs/software.eessi.io/${fn}" > ${outfile}
    +                grep "/cvmfs/software.eessi.io/${fn}$" $outfile
    
                   # test use of --resume
                   elif [[ ${{matrix.SCRIPT_TEST}} == 'resume' ]]; then
    @@ -120,12 +120,12 @@ jobs:
                   elif [[ ${{matrix.SCRIPT_TEST}} == 'save' ]]; then
                     outfile=out_save.txt
                     fn="test_${RANDOM}.txt"
    -                test_cmd="touch /cvmfs/pilot.nessi.no/${fn}"
    +                test_cmd="touch /cvmfs/software.eessi.io/${fn}"
                     ./eessi_container.sh --verbose --mode shell --access rw --save test-save.tar <<< "${test_cmd}" 2>&1 | tee ${outfile}
                     rm -f ${outfile}
    
    -                ./eessi_container.sh --verbose --mode shell --access rw --resume test-save.tar <<< "ls -l /cvmfs/pilot.nessi.no/${fn}" > ${outfile}
    -                grep "/cvmfs/pilot.nessi.no/${fn}$" $outfile
    +                ./eessi_container.sh --verbose --mode shell --access rw --resume test-save.tar <<< "ls -l /cvmfs/software.eessi.io/${fn}" > ${outfile}
    +                grep "/cvmfs/software.eessi.io/${fn}$" $outfile
    
                     tar tfv test-save.tar | grep "overlay-upper/${fn}"
    

  • .github/workflows/tests_archdetect.yml

    diff

    diff --git a/.github/workflows/tests_archdetect.yml b/.github/workflows/tests_archdetect.yml
    index 74dbf03..1e8b830 100644
    --- a/.github/workflows/tests_archdetect.yml
    +++ b/.github/workflows/tests_archdetect.yml
    @@ -13,23 +13,23 @@ jobs:
               - x86_64/intel/skylake_avx512/archspec-linux-6132
               - x86_64/amd/zen2/Azure-CentOS7-7V12
               - x86_64/amd/zen3/Azure-CentOS7-7V73X
    -          # commented out since these targets are currently not supported in pilot.nessi.no repo
    +          - aarch64/neoverse_n1/Azure-Ubuntu20-Altra
    +          - aarch64/neoverse_n1/AWS-awslinux-graviton2
    +          - aarch64/neoverse_v1/AWS-awslinux-graviton3
    +          # commented out since these targets are currently not supported in software.eessi.io repo
               # (and some tests assume that the corresponding subdirectory in software layer is there)
               # - ppc64le/power9le/unknown-power9le
    -          # - aarch64/neoverse-n1/Azure-Ubuntu20-Altra
    -          # - aarch64/neoverse-n1/AWS-awslinux-graviton2
    -          # - aarch64/neoverse-v1/AWS-awslinux-graviton3
           fail-fast: false
         steps:
         - name: checkout
           uses: actions/checkout@93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 # v3.1.0
    
    -    - name: Mount NESSI CernVM-FS repository
    +    - name: Mount EESSI CernVM-FS pilot repository
           uses: cvmfs-contrib/github-action-cvmfs@d4641d0d591c9a5c3be23835ced2fb648b44c04b # v3.1
           with:
    -          cvmfs_config_package: https://github.com/NorESSI/filesystem-layer/releases/download/latest/cvmfs-config-nessi_latest_all.deb
    +          cvmfs_config_package: https://github.com/EESSI/filesystem-layer/releases/download/latest/cvmfs-config-eessi_latest_all.deb
               cvmfs_http_proxy: DIRECT
    -          cvmfs_repositories: pilot.nessi.no
    +          cvmfs_repositories: software.eessi.io
    
         - name: test eessi_archdetect.sh
           run: |

  • EESSI-install-software.sh

    diff

    diff --git a/EESSI-install-software.sh b/EESSI-install-software.sh
    index eb616b1..69de9d1 100755
    --- a/EESSI-install-software.sh
    +++ b/EESSI-install-software.sh
    @@ -1,6 +1,6 @@
     #!/bin/bash
     #
    -# Script to install NESSI software stack (version set through init/eessi_defaults)
    +# Script to install EESSI software stack (version set through init/eessi_defaults)
    
     # see example parsing of command line arguments at
     #   https://wiki.bash-hackers.org/scripting/posparams#using_a_while_loop

  • README.md

    diff

    diff --git a/README.md b/README.md
    index 4f06139..ab73ad7 100644
    --- a/README.md
    +++ b/README.md
    @@ -1,25 +1,25 @@
     # Software layer
    
    -The software layer of the NESSI project uses [EasyBuild](https://docs.easybuild.io), [Lmod](https://lmod.readthedocs.io) and [archspec](https://archspec.readthedocs.io).
    +The software layer of the EESSI project uses [EasyBuild](https://docs.easybuild.io), [Lmod](https://lmod.readthedocs.io) and [archspec](https://archspec.readthedocs.io).
    
     See also https://www.eessi.io/docs/software_layer .
    
    -## Accessing the software stack
    +## Pilot software stack
    
     You can set up your environment by sourcing the init script:
    

    -$ source /cvmfs/pilot.nessi.no/versions/2023.06/init/bash
    -Found NESSI repo @ /cvmfs/pilot.nessi.no/versions/2023.06!
    -Derived subdirectory for software layer: x86_64/intel/broadwell
    +$ source /cvmfs/software.eessi.io/versions/2023.06/init/bash
    +Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
    +Derived subdirectory for software layer: x86_64/intel/haswell
    Using x86_64/intel/haswell subdirectory for software layer
    Initializing Lmod...
    -Prepending /cvmfs/pilot.nessi.no/versions/2023.06/software/x86_64/intel/broadwell/modules/all to $MODULEPATH...
    -Environment set up to use NESSI software stack, have fun!
    -{NESSI 2023.06} $
    +Prepending /cvmfs/software.eessi.io/versions/2023.06/software/x86_64/intel/haswell/modules/all to $MODULEPATH...
    +Environment set up to use EESSI (2023.06), have fun!
    +[EESSI 2023.06] $

    
    -### Accessing NESSI via a container
    +### Accessing EESSI via a container
    
    You need Singularity version 3.7 or newer. Then, simply run
    
    @@ -40,6 +40,6 @@ Note, not all features/arguments listed via `./eessi_container.sh --help` are im
    The software in this repository is distributed under the terms of the
    [GNU General Public License v2.0](https://opensource.org/licenses/GPL-2.0).
    
    -See [LICENSE](https://github.com/NorESSI/software-layer/blob/main/LICENSE) for more information.
    +See [LICENSE](https://github.com/EESSI/software-layer/blob/main/LICENSE) for more information.
    
    SPDX-License-Identifier: GPL-2.0-only
    

  • bot/build.sh

    diff

    diff --git a/bot/build.sh b/bot/build.sh
    index 7eb15f3..1622e75 100755
    --- a/bot/build.sh
    +++ b/bot/build.sh
    @@ -128,7 +128,7 @@ EESSI_REPOS_CFG_DIR_OVERRIDE=$(cfg_get_value "repository" "repos_cfg_dir")
     export EESSI_REPOS_CFG_DIR_OVERRIDE=${EESSI_REPOS_CFG_DIR_OVERRIDE:-${PWD}/cfg}
     echo "bot/build.sh: EESSI_REPOS_CFG_DIR_OVERRIDE='${EESSI_REPOS_CFG_DIR_OVERRIDE}'"
    
    -# determine NESSI version to be used from .repository.repo_version in ${JOB_CFG_FILE}
    +# determine EESSI version to be used from .repository.repo_version in ${JOB_CFG_FILE}
     # here, just set & export EESSI_VERSION_OVERRIDE
     # next script (eessi_container.sh) makes use of it via sourcing init scripts
     # (e.g., init/eessi_defaults or init/minimal_eessi_env)

  • build_container.sh

    diff

    diff --git a/build_container.sh b/build_container.sh
    new file mode 100755
    index 0000000..23a9e66
    --- /dev/null
    +++ b/build_container.sh
    @@ -0,0 +1,69 @@
    +#!/bin/bash
    +
    +base_dir=$(dirname $(realpath $0))
    +
    +BUILD_CONTAINER="docker://ghcr.io/eessi/build-node:debian11"
    +
    +if [ $# -lt 2 ]; then
    +    echo "Usage: $0 <shell|run> <path for temporary directories>" >&2
    +    exit 1
    +fi
    +SHELL_OR_RUN=$1
    +EESSI_TMPDIR=$2
    +shift 2
    +
    +if [ "$SHELL_OR_RUN" == "run" ] && [ $# -eq 0 ]; then
    +    echo "ERROR: No command specified to run?!" >&2
    +    exit 1
    +fi
    +
    +# make sure specified temporary directory exists
    +mkdir -p $EESSI_TMPDIR
    +
    +echo "Using $EESSI_TMPDIR as parent for temporary directories..."
    +
    +# create temporary directories
    +mkdir -p $EESSI_TMPDIR/{home,overlay-upper,overlay-work}
    +mkdir -p $EESSI_TMPDIR/{var-lib-cvmfs,var-run-cvmfs}
    +# configure Singularity
    +export SINGULARITY_CACHEDIR=$EESSI_TMPDIR/singularity_cache
    +
    +# take into account that $SINGULARITY_BIND may be defined already, to bind additional paths into the build container
    +BIND_PATHS="$EESSI_TMPDIR/var-run-cvmfs:/var/run/cvmfs,$EESSI_TMPDIR/var-lib-cvmfs:/var/lib/cvmfs,$EESSI_TMPDIR"
    +if [ -z $SINGULARITY_BIND ]; then
    +    export SINGULARITY_BIND="$BIND_PATHS"
    +else
    +    export SINGULARITY_BIND="$SINGULARITY_BIND,$BIND_PATHS"
    +fi
    +
    +# allow that SINGULARITY_HOME is defined before script is run
    +if [ -z $SINGULARITY_HOME ]; then
    +    export SINGULARITY_HOME="$EESSI_TMPDIR/home:/home/$USER"
    +fi
    +
    +source ${base_dir}/init/eessi_defaults
    +# strip "/cvmfs/" from default setting
    +repo_name=${EESSI_CVMFS_REPO/\/cvmfs\//}
    +
    +# set environment variables for fuse mounts in Singularity container
    +export EESSI_PILOT_READONLY="container:cvmfs2 ${repo_name} /cvmfs_ro/${repo_name}"
    +export EESSI_PILOT_WRITABLE_OVERLAY="container:fuse-overlayfs -o lowerdir=/cvmfs_ro/${repo_name} -o upperdir=$EESSI_TMPDIR/overlay-upper -o workdir=$EESSI_TMPDIR/overlay-work ${EESSI_CVMFS_REPO}"
    +
    +# pass $EESSI_SOFTWARE_SUBDIR_OVERRIDE into build container (if set)
    +if [ ! -z ${EESSI_SOFTWARE_SUBDIR_OVERRIDE} ]; then
    +    export SINGULARITYENV_EESSI_SOFTWARE_SUBDIR_OVERRIDE=${EESSI_SOFTWARE_SUBDIR_OVERRIDE}
    +    # also specify via $APPTAINERENV_* (future proof, cfr. https://apptainer.org/docs/user/latest/singularity_compatibility.html#singularity-environment-variable-compatibility)
    +    export APPTAINERENV_EESSI_SOFTWARE_SUBDIR_OVERRIDE=${EESSI_SOFTWARE_SUBDIR_OVERRIDE}
    +fi
    +
    +if [ "$SHELL_OR_RUN" == "shell" ]; then
    +    # start shell in Singularity container, with EESSI repository mounted with writable overlay
    +    echo "Starting Singularity build container..."
    +    singularity shell --fusemount "$EESSI_PILOT_READONLY" --fusemount "$EESSI_PILOT_WRITABLE_OVERLAY" $BUILD_CONTAINER
    +elif [ "$SHELL_OR_RUN" == "run" ]; then
    +    echo "Running '$@' in Singularity build container..."
    +    singularity exec --fusemount "$EESSI_PILOT_READONLY" --fusemount "$EESSI_PILOT_WRITABLE_OVERLAY" $BUILD_CONTAINER "$@"
    +else
    +    echo "ERROR: Unknown action specified: $SHELL_OR_RUN (should be either 'shell' or 'run')" >&2
    +    exit 1
    +fi

  • check_missing_installations.sh

    diff

    diff --git a/check_missing_installations.sh b/check_missing_installations.sh
    index dd292e4..5ea7c5a 100755
    --- a/check_missing_installations.sh
    +++ b/check_missing_installations.sh
    @@ -1,6 +1,6 @@
     #!/bin/bash
     #
    -# Script to check for missing installations in NESSI software stack (version 2023.06)
    +# Script to check for missing installations in EESSI software stack
     #
     # author: Kenneth Hoste (@boegel)
     # author: Thomas Roeblitz (@trz42)

  • configure_easybuild

    diff

    diff --git a/configure_easybuild b/configure_easybuild
    index c1bd1d3..c67b879 100644
    --- a/configure_easybuild
    +++ b/configure_easybuild
    @@ -25,7 +25,6 @@ fi
    
     # note: filtering Bison may break some installations, like Qt5 (see https://github.com/EESSI/software-layer/issues/49)
     # filtering pkg-config breaks R-bundle-Bioconductor installation (see also https://github.com/easybuilders/easybuild-easyconfigs/pull/11104)
    -# problems occur when filtering pkg-config with gnuplot too (picks up Lua 5.1 from $EPREFIX rather than from Lua 5.3 dependency)
     DEPS_TO_FILTER=Autoconf,Automake,Autotools,binutils,bzip2,DBus,flex,gettext,gperf,help2man,intltool,libreadline,libtool,M4,makeinfo,ncurses,util-linux,XZ,zlib
     # For aarch64 we need to also filter out Yasm.
     # See https://github.com/easybuilders/easybuild-easyconfigs/issues/11190

  • create_directory_tarballs.sh

    diff

    diff --git a/create_directory_tarballs.sh b/create_directory_tarballs.sh
    index a1f05b5..0270719 100755
    --- a/create_directory_tarballs.sh
    +++ b/create_directory_tarballs.sh
    @@ -9,7 +9,7 @@ fi
    
     version=$1
    
    -SOFTWARE_LAYER_TARBALL_URL=https://github.com/NorESSI/software-layer/tarball/nessi.no-2023.06
    +SOFTWARE_LAYER_TARBALL_URL="https://github.com/EESSI/software-layer/tarball/${version}-software.eessi.io"
    
     TOPDIR=$(dirname $(realpath $0))
    

  • create_tarball.sh

    diff

    diff --git a/create_tarball.sh b/create_tarball.sh
    index f6239cf..a619df9 100755
    --- a/create_tarball.sh
    +++ b/create_tarball.sh
    @@ -43,7 +43,7 @@ module_files_list=${tmpdir}/module_files.list.txt
     if [ -d ${eessi_version}/software/${os}/${cpu_arch_subdir}/.lmod ]; then
         # include Lmod cache and configuration file (lmodrc.lua),
         # skip whiteout files and backup copies of Lmod cache (spiderT.old.*)
    -    find ${eessi_version}/software/${os}/${cpu_arch_subdir}/.lmod -type f | egrep -v '/\.wh\.|spiderT.old' > ${files_list}
    +    find ${eessi_version}/software/${os}/${cpu_arch_subdir}/.lmod -type f | egrep -v '/\.wh\.|spiderT.old' >> ${files_list}
     fi
    
     # include scripts that were copied by install_scripts.sh, which we want to ship in EESSI repository

  • easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021a.yml

    diff

    diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021a.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021a.yml
    deleted file mode 100644
    index 28612c5..0000000
    --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021a.yml
    +++ /dev/null
    @@ -1,27 +0,0 @@
    -easyconfigs:
    -  - OpenSSL-1.1.eb
    -  # - OpenSSL-1.1.eb:
    -  #     options:
    -  #       include-easyblocks-from-pr: 2922
    -  - git-2.32.0-GCCcore-10.3.0-nodocs.eb
    -  - GCC-10.3.0
    -  - CMake-3.20.1-GCCcore-10.3.0.eb
    -  # - CMake-3.20.1-GCCcore-10.3.0.eb:
    -  #     options:
    -  #       include-easyblocks-from-pr: 2248
    -  - Rust-1.52.1-GCCcore-10.3.0.eb
    -  - foss-2021a.eb
    -  - QuantumESPRESSO-6.7-foss-2021a.eb
    -  - GROMACS-2021.3-foss-2021a.eb:
    -       options:
    -         download-timeout: 1000
    -  - libGLU-9.0.1-GCCcore-10.3.0.eb
    -  # since the BAGEL ec file is new we have to keep the from-pr or eb fails
    -  - BAGEL-1.2.2-foss-2021a.eb:
    -        options:
    -          from-pr: 18446
    -  - WRF-4.3-foss-2021a-dmpar.eb
    -  - Yambo-5.1.1-foss-2021a.eb:
    -        options:
    -          from-pr: 18905
    -

  • easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021b.yml

    diff

    diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021b.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021b.yml
    deleted file mode 100644
    index d838e15..0000000
    --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021b.yml
    +++ /dev/null
    @@ -1,18 +0,0 @@
    -easyconfigs:
    -  - GCC-11.2.0
    -  - git-2.33.1-GCCcore-11.2.0-nodocs.eb
    -  - CMake-3.21.1-GCCcore-11.2.0.eb
    -  # - CMake-3.21.1-GCCcore-11.2.0.eb:
    -  #     options:
    -  #       include-easyblocks-from-pr: 2248
    -  - OpenMPI-4.1.1-GCC-11.2.0.eb
    -  - FFTW-3.3.10-gompi-2021b.eb
    -  - BLIS-0.8.1-GCC-11.2.0.eb
    -  - foss-2021b.eb
    -  # note, the below CMake has never been installed, just put here to use the
    -  # same version as easy does
    -  # - CMake-3.22.1-GCCcore-11.2.0.eb:
    -  #     options:
    -  #       include-easyblocks-from-pr: 2248
    -  # - zlib-1.2.11-GCCcore-11.2.0
    -  - QuantumESPRESSO-6.8-foss-2021b.eb

  • easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022a.yml

    diff

    diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022a.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022a.yml
    deleted file mode 100644
    index 9991de1..0000000
    --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022a.yml
    +++ /dev/null
    @@ -1,12 +0,0 @@
    -easyconfigs:
    -  - GCC-11.3.0
    -  - git-2.36.0-GCCcore-11.3.0-nodocs.eb
    -  - CMake-3.23.1-GCCcore-11.3.0.eb
    -  # - CMake-3.23.1-GCCcore-11.3.0.eb:
    -  #     options:
    -  #       include-easyblocks-from-pr: 2248
    -  - BLIS-0.9.0-GCC-11.3.0.eb
    -  - OpenMPI-4.1.4-GCC-11.3.0.eb
    -  - FFTW.MPI-3.3.10-gompi-2022a.eb
    -  - foss-2022a.eb
    -  - Python-3.10.4-GCCcore-11.3.0.eb

  • easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022b.yml

    diff

    diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022b.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022b.yml
    deleted file mode 100644
    index 129b709..0000000
    --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022b.yml
    +++ /dev/null
    @@ -1,12 +0,0 @@
    -easyconfigs:
    -  - GCC-12.2.0
    -  - git-2.38.1-GCCcore-12.2.0-nodocs.eb
    -  - CMake-3.24.3-GCCcore-12.2.0.eb
    -  # - CMake-3.24.3-GCCcore-12.2.0.eb:
    -  #     options:
    -  #       include-easyblocks-from-pr: 2248
    -  - BLIS-0.9.0-GCC-12.2.0.eb
    -  - OpenMPI-4.1.4-GCC-12.2.0.eb
    -  - FFTW.MPI-3.3.10-gompi-2022b.eb
    -  - Python-3.10.8-GCCcore-12.2.0.eb
    -  - foss-2022b.eb

  • easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-system.yml

    diff

    diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-system.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-system.yml
    deleted file mode 100644
    index a5f84c7..0000000
    --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-system.yml
    +++ /dev/null
    @@ -1,4 +0,0 @@
    -easyconfigs:
    -  - EasyBuild-4.8.0.eb:
    -      options:
    -        from-pr: 18282

  • easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.0-system.yml

    diff

    diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.0-system.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.0-system.yml
    deleted file mode 100644
    index 96d79de..0000000
    --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.0-system.yml
    +++ /dev/null
    @@ -1,4 +0,0 @@
    -easyconfigs:
    -  - EasyBuild-4.8.1.eb:
    -      options:
    -        from-pr: 18761

  • easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-2022a.yml

    diff

    diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-2022a.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-2022a.yml
    deleted file mode 100644
    index 14f936b..0000000
    --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-2022a.yml
    +++ /dev/null
    @@ -1,66 +0,0 @@
    -easyconfigs:
    -  - BAMM-2.5.0-foss-2022a.eb
    -  - BamTools-2.5.2-GCC-11.3.0.eb
    -  - BCFtools-1.15.1-GCC-11.3.0.eb
    -  - Bowtie2-2.4.5-GCC-11.3.0.eb
    -  - FastTree-2.1.11-GCCcore-11.3.0.eb
    -  - SAMtools-1.16.1-GCC-11.3.0.eb
    -  - gzip-1.12-GCCcore-11.3.0.eb
    -  - lz4-1.9.3-GCCcore-11.3.0.eb
    -  - zstd-1.5.2-GCCcore-11.3.0.eb
    -  - Pillow-9.1.1-GCCcore-11.3.0.eb:
    -      # avoid that hardcoded paths like /usr/include are used in build commands
    -        options:
    -          from-pr: 18881
    -  - Boost.Python-1.79.0-GCC-11.3.0.eb
    -  - netCDF-4.9.0-gompi-2022a.eb:
    -      # use updated CMakeMake easyblock to avoid that -DCMAKE_SKIP_RPATH=ON is used, which breaks the netCDF test step
    -      # see https://github.com/easybuilders/easybuild-easyblocks/pull/3012
    -      options:
    -        include-easyblocks-from-pr: 3012
    -  - NCO-5.1.0-foss-2022a.eb
    -  - AdapterRemoval-2.3.3-GCC-11.3.0.eb
    -  - BEDTools-2.30.0-GCC-11.3.0.eb
    -  - Pysam-0.20.0-GCC-11.3.0.eb
    -  - TransDecoder-5.5.0-GCC-11.3.0.eb
    -  - VCFtools-0.1.16-GCC-11.3.0.eb
    -  - XML-LibXML-2.0207-GCCcore-11.3.0.eb
    -  - CMake-3.24.3-GCCcore-11.3.0.eb
    -  - elfutils-0.187-GCCcore-11.3.0.eb
    -  - Ninja-1.10.2-GCCcore-11.3.0.eb
    -  - Z3-4.10.2-GCCcore-11.3.0.eb
    -  - SciPy-bundle-2022.05-foss-2022a.eb
    -  - Xvfb-21.1.3-GCCcore-11.3.0.eb:
    -      # enable exec permissions for xvfb-run after copying;
    -      # need to also enable user write permissions on xvfb-run to ensure that copying with preserved permissions works
    -      options:
    -        from-pr: 18834
    -  - tbb-2021.5.0-GCCcore-11.3.0.eb
    -  - CMSeq-1.0.4-foss-2022a.eb
    -  - Pango-1.50.7-GCCcore-11.3.0.eb
    -  - nodejs-16.15.1-GCCcore-11.3.0.eb:
    -      options:
    -        download-timeout: 1000
    -  - ecCodes-2.27.0-gompi-2022a.eb:
    -        options:
    -          download-timeout: 1000
    -  - GLPK-5.0-GCCcore-11.3.0.eb
    -  - libgit2-1.4.3-GCCcore-11.3.0.eb
    -  - libGLU-9.0.2-GCCcore-11.3.0.eb
    -  - libsndfile-1.1.0-GCCcore-11.3.0.eb
    -  - LibTIFF-4.3.0-GCCcore-11.3.0.eb
    -  - MPFR-4.1.0-GCCcore-11.3.0.eb
    -  - NLopt-2.7.1-GCCcore-11.3.0.eb
    -  - PCRE2-10.40-GCCcore-11.3.0.eb
    -  - Tk-8.6.12-GCCcore-11.3.0.eb
    -  - GROMACS-2023.1-foss-2022a.eb
    -  - MUMPS-5.5.1-foss-2022a-metis.eb
    -  - Pillow-9.1.1-GCCcore-11.3.0.eb:
    -      # Uses a custom hook since has zlib as dependency which has hard coded header and library path within Pillow code.
    -      options:
    -        from-pr: 18881
    -  - ParaView-5.10.1-foss-2022a-mpi.eb:
    -        options:
    -          download-timeout: 1000
    -  - ASE-3.22.1-foss-2022a.eb
    -  - HISAT2-2.2.1-gompi-2022a.eb

  • easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.2-2022a.yml

    diff

    diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.2-2022a.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.2-2022a.yml
    deleted file mode 100644
    index f191e76..0000000
    --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.2-2022a.yml
    +++ /dev/null
    @@ -1,3 +0,0 @@
    -easyconfigs:
    -  - CDO-2.0.6-gompi-2022a.eb
    -  - OpenBabel-3.1.1-gompi-2022a.eb

  • easystacks/pilot.nessi.no/2023.06/README.md

    diff

    diff --git a/easystacks/pilot.nessi.no/2023.06/README.md b/easystacks/software.eessi.io/2023.06/README.md
    similarity index 100%
    rename from easystacks/pilot.nessi.no/2023.06/README.md
    rename to easystacks/software.eessi.io/2023.06/README.md

  • easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-system.yml

    diff

    diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-system.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-001-system.yml
    similarity index 54%
    rename from easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-system.yml
    rename to easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-001-system.yml
    index 7418c0a..f02b9f2 100644
    --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-system.yml
    +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-001-system.yml
    @@ -1,6 +1,7 @@
     easyconfigs:
    -  # wraps around Java/11.0.20
    -  - Java-11.eb
       - EasyBuild-4.8.2.eb:
           options:
             from-pr: 19105
    +  - Nextflow-23.10.0.eb:
    +      options:
    +        from-pr: 19172

  • easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-010-CUDA.yml

    diff

    diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-010-CUDA.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-010-CUDA.yml
    new file mode 100644
    index 0000000..dda274b
    --- /dev/null
    +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-010-CUDA.yml
    @@ -0,0 +1,5 @@
    +easyconfigs:
    +  - CUDA-12.1.1.eb:
    +      options:
    +        include-easyblocks-from-pr: 3045
    +        accept-eula-for: CUDA

  • easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2022b.yml

    diff

    diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2022b.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2022b.yml
    new file mode 100644
    index 0000000..69ffb75
    --- /dev/null
    +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2022b.yml
    @@ -0,0 +1,6 @@
    +easyconfigs:
    +  - foss-2022b.eb
    +  - HarfBuzz-5.3.1-GCCcore-12.2.0.eb:
    +      options:
    +        from-pr: 19339
    +  - Qt5-5.15.7-GCCcore-12.2.0.eb

  • easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2023a.yml

    diff

    diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2023a.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2023a.yml
    new file mode 100644
    index 0000000..aab910d
    --- /dev/null
    +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2023a.yml
    @@ -0,0 +1,48 @@
    +easyconfigs:
    +  - GCC-12.3.0.eb
    +  - Rust-1.70.0-GCCcore-12.3.0.eb:
    +      # fix build of Rust 1.70.0 by disabling download of pre-built LLVM;
    +      # see https://github.com/easybuilders/easybuild-easyblocks/pull/3038
    +      options:
    +        include-easyblocks-from-pr: 3038
    +  - foss-2023a.eb
    +  - pybind11-2.11.1-GCCcore-12.3.0.eb:
    +      # avoid indirect dependency on old CMake version built with GCCcore/10.2.0 via Catch2 build dependency;
    +      # see https://github.com/easybuilders/easybuild-easyconfigs/pull/19270
    +      options:
    +        from-pr: 19270
    +  - SciPy-bundle-2023.07-gfbf-2023a.eb
    +  - TensorFlow-2.13.0-foss-2023a.eb:
    +      # patch setup.py for grpcio extension in TensorFlow 2.13.0 easyconfigs to take into account alternate sysroot;
    +      # see https://github.com/easybuilders/easybuild-easyconfigs/pull/19268
    +      options:
    +        from-pr: 19268
    +  - X11-20230603-GCCcore-12.3.0.eb
    +  - HarfBuzz-5.3.1-GCCcore-12.3.0.eb:
    +      options:
    +        from-pr: 19339
    +  - Qt5-5.15.10-GCCcore-12.3.0.eb
    +  - OSU-Micro-Benchmarks-7.1-1-gompi-2023a.eb
    +  - LHAPDF-6.5.4-GCC-12.3.0.eb:
    +      options:
    +        from-pr: 19363
    +  - LoopTools-2.15-GCC-12.3.0.eb:
    +      options:
    +        from-pr: 19397
    +  - R-4.3.2-gfbf-2023a.eb:
    +      options:
    +        from-pr: 19185
    +  - Boost-1.82.0-GCC-12.3.0.eb
    +  - netCDF-4.9.2-gompi-2023a.eb
    +  - FFmpeg-6.0-GCCcore-12.3.0.eb
    +  - CUDA-Samples-12.1-GCC-12.3.0-CUDA-12.1.1.eb:
    +      # use easyconfig that only install subset of CUDA samples,
    +      # to circumvent problem with nvcc linking to glibc of host OS,
    +      # see https://github.com/easybuilders/easybuild-easyconfigs/pull/19189;
    +      # and where additional samples are excluded because they fail to build on aarch64,
    +      # see https://github.com/easybuilders/easybuild-easyconfigs/pull/19451;
    +      options:
    +        from-pr: 19451
    +  - ALL-0.9.2-foss-2023a.eb:
    +      options:
    +        from-pr: 19455

  • easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-001-system.yml

    diff

    diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-001-system.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-001-system.yml
    new file mode 100644
    index 0000000..25c13e4
    --- /dev/null
    +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-001-system.yml
    @@ -0,0 +1,5 @@
    +easyconfigs:
    +  - EasyBuild-4.9.0.eb:
    +      options:
    +        from-pr: 19464
    +  - ReFrame-4.3.3.eb

  • easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2022b.yml

    diff

    diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2022b.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2022b.yml
    new file mode 100644
    index 0000000..fdacd95
    --- /dev/null
    +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2022b.yml
    @@ -0,0 +1,2 @@
    +easyconfigs:
    +  - SciPy-bundle-2023.02-gfbf-2022b.eb

  • easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023a.yml

    diff

    diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023a.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023a.yml
    new file mode 100644
    index 0000000..eba776a
    --- /dev/null
    +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023a.yml
    @@ -0,0 +1,10 @@
    +easyconfigs:
    +  - OpenFOAM-11-foss-2023a.eb:
    +      # see https://github.com/easybuilders/easybuild-easyconfigs/pull/19545
    +      options:
    +        from-pr: 19545
    +  - at-spi2-core-2.49.91-GCCcore-12.3.0.eb
    +  - ESPResSo-4.2.1-foss-2023a.eb:
    +      # see https://github.com/easybuilders/easybuild-easyconfigs/pull/19592
    +      options:
    +        from-pr: 19592

  • easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023b.yml

    diff

    diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023b.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023b.yml
    new file mode 100644
    index 0000000..9a3b8ef
    --- /dev/null
    +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023b.yml
    @@ -0,0 +1,7 @@
    +easyconfigs:
    +  - GCC-13.2.0.eb
    +  - foss-2023b.eb
    +  - SciPy-bundle-2023.11-gfbf-2023b.eb
    +  - netCDF-4.9.2-gompi-2023b.eb:
    +      options:
    +        from-pr: 19534

  • eb_hooks.py

    diff

    diff --git a/eb_hooks.py b/eb_hooks.py
    index b3cbc4b..2bc2fde 100644
    --- a/eb_hooks.py
    +++ b/eb_hooks.py
    @@ -185,20 +185,21 @@ def parse_hook_fontconfig_add_fonts(ec, eprefix):
    
    
     def parse_hook_openblas_relax_lapack_tests_num_errors(ec, eprefix):
    -    """Relax number of failing numerical LAPACK tests for aarch64/neoverse_v1 CPU target."""
    +    """Relax number of failing numerical LAPACK tests for aarch64/neoverse_v1 CPU target for OpenBLAS < 0.3.23"""
         cpu_target = get_eessi_envvar('EESSI_SOFTWARE_SUBDIR')
         if ec.name == 'OpenBLAS':
    -        # relax maximum number of failed numerical LAPACK tests for aarch64/neoverse_v1 CPU target
    -        # since the default setting of 150 that works well on other aarch64 targets and x86_64 is a bit too strict
    -        # See https://github.com/EESSI/software-layer/issues/314
    -        cfg_option = 'max_failing_lapack_tests_num_errors'
    -        if cpu_target == CPU_TARGET_NEOVERSE_V1:
    -            orig_value = ec[cfg_option]
    -            ec[cfg_option] = 400
    -            print_msg("Maximum number of failing LAPACK tests with numerical errors for %s relaxed to %s (was %s)",
    -                      ec.name, ec[cfg_option], orig_value)
    -        else:
    -            print_msg("Not changing option %s for %s on non-AARCH64", cfg_option, ec.name)
    +        if LooseVersion(ec.version) < LooseVersion('0.3.23'):
    +            # relax maximum number of failed numerical LAPACK tests for aarch64/neoverse_v1 CPU target
    +            # since the default setting of 150 that works well on other aarch64 targets and x86_64 is a bit too strict
    +            # See https://github.com/EESSI/software-layer/issues/314
    +            cfg_option = 'max_failing_lapack_tests_num_errors'
    +            if cpu_target == CPU_TARGET_NEOVERSE_V1:
    +                orig_value = ec[cfg_option]
    +                ec[cfg_option] = 400
    +                print_msg("Maximum number of failing LAPACK tests with numerical errors for %s relaxed to %s (was %s)",
    +                          ec.name, ec[cfg_option], orig_value)
    +            else:
    +                print_msg("Not changing option %s for %s on non-AARCH64", cfg_option, ec.name)
         else:
             raise EasyBuildError("OpenBLAS-specific hook triggered for non-OpenBLAS easyconfig?!")
    
    @@ -393,6 +394,18 @@ def pre_test_hook_ignore_failing_tests_SciPybundle(self, *args, **kwargs):
         if self.name == 'SciPy-bundle' and self.version in scipy_bundle_versions and cpu_target == CPU_TARGET_NEOVERSE_V1:
             self.cfg['testopts'] = "|| echo ignoring failing tests"
    
    +def pre_test_hook_ignore_failing_tests_netCDF(self, *args, **kwargs):
    +    """
    +    Pre-test hook for netCDF: skip failing tests for selected netCDF versions on neoverse_v1
    +    cfr. https://github.com/EESSI/software-layer/issues/425
    +    The following tests are problematic:
    +        163 - nc_test4_run_par_test (Timeout)
    +        190 - h5_test_run_par_tests (Timeout)
    +    A few other tests are skipped in the easyconfig and patches for similar issues, see above issue for details.
    +    """
    +    cpu_target = get_eessi_envvar('EESSI_SOFTWARE_SUBDIR')
    +    if self.name == 'netCDF' and self.version == '4.9.2' and cpu_target == CPU_TARGET_NEOVERSE_V1:
    +        self.cfg['testopts'] = "|| echo ignoring failing tests"
    
     def pre_single_extension_hook(ext, *args, **kwargs):
         """Main pre-extension: trigger custom functions based on software name."""
    @@ -573,6 +586,7 @@ PRE_TEST_HOOKS = {
         'ESPResSo': pre_test_hook_ignore_failing_tests_ESPResSo,
         'FFTW.MPI': pre_test_hook_ignore_failing_tests_FFTWMPI,
         'SciPy-bundle': pre_test_hook_ignore_failing_tests_SciPybundle,
    +    'netCDF': pre_test_hook_ignore_failing_tests_netCDF,
     }
    
     PRE_SINGLE_EXTENSION_HOOKS = {

  • eessi-2023.06-known-issues.yml

    diff

    diff --git a/eessi-2023.06-known-issues.yml b/eessi-2023.06-known-issues.yml
    new file mode 100644
    index 0000000..475ee2c
    --- /dev/null
    +++ b/eessi-2023.06-known-issues.yml
    @@ -0,0 +1,28 @@
    +- aarch64/neoverse_v1:
    +  - ESPResSo-4.2.1-foss-2023a:
    +      - issue: https://github.com/EESSI/software-layer/issues/363
    +      - info: "ESPResSo tests failing due to timeouts"
    +  - FFTW.MPI-3.3.10-gompi-2023a:
    +      - issue: https://github.com/EESSI/software-layer/issues/325
    +      - info: "Flaky FFTW tests, random failures"
    +  - FFTW.MPI-3.3.10-gompi-2023b:
    +      - issue: https://github.com/EESSI/software-layer/issues/325
    +      - info: "Flaky FFTW tests, random failures"
    +  - netCDF-4.9.2-gompi-2023a.eb:
    +      - issue: https://github.com/EESSI/software-layer/issues/425
    +      - info: "netCDF intermittent test failures"
    +  - netCDF-4.9.2-gompi-2023b.eb:
    +      - issue: https://github.com/EESSI/software-layer/issues/425
    +      - info: "netCDF intermittent test failures"
    +  - OpenBLAS-0.3.21-GCC-12.2.0:
    +    - issue: https://github.com/EESSI/software-layer/issues/314
    +    - info: "Increased number of numerical errors in OpenBLAS test suite (344 vs max. 150 on x86_64/*)"
    +  - SciPy-bundle-2023.02-gfbf-2022b:
    +    - issue: https://github.com/EESSI/software-layer/issues/318
    +    - info: "numpy built with -march=armv8.4-a instead of -mcpu=native (no SVE) + 2 failing tests (vs 50005 passed) in scipy test suite"
    +  - SciPy-bundle-2023.07-gfbf-2023a:
    +    - issue: https://github.com/EESSI/software-layer/issues/318
    +    - info: "2 failing tests (vs 54409 passed) in scipy test suite"
    +  - SciPy-bundle-2023.11-gfbf-2023b:
    +    - issue: https://github.com/EESSI/software-layer/issues/318
    +    - info: "2 failing tests (vs 54876 passed) in scipy test suite"

  • eessi_container.sh

    diff

    diff --git a/eessi_container.sh b/eessi_container.sh
    index bfc5963..d6e9558 100755
    --- a/eessi_container.sh
    +++ b/eessi_container.sh
    @@ -30,8 +30,8 @@
     # -. initial settings & exit codes
     TOPDIR=$(dirname $(realpath $0))
    
    -source ${TOPDIR}/scripts/utils.sh
    -source ${TOPDIR}/scripts/cfg_files.sh
    +source "${TOPDIR}"/scripts/utils.sh
    +source "${TOPDIR}"/scripts/cfg_files.sh
    
     # exit codes: bitwise shift codes to allow for combination of exit codes
     # ANY_ERROR_EXITCODE is sourced from ${TOPDIR}/scripts/utils.sh
    @@ -83,7 +83,7 @@ display_help() {
       echo "                           MODE==run (run a script or command) [default: shell]"
       echo "  -n | --nvidia MODE     - configure the container to work with NVIDIA GPUs,"
       echo "                           MODE==install for a CUDA installation, MODE==run to"
    -  echo "                           attach a GPU, MODE==all for both [default: false]"
    +  echo "                           attach a GPU, MODE==all for both [default: false]"
       echo "  -r | --repository CFG  - configuration file or identifier defining the"
       echo "                           repository to use [default: EESSI via"
       echo "                           default container, see --container]"
    @@ -164,7 +164,7 @@ while [[ $# -gt 0 ]]; do
           SETUP_NVIDIA=1
           NVIDIA_MODE="$2"
           shift 2
    -      ;;
    +      ;;
         -r|--repository)
           REPOSITORY="$2"
           shift 2
    @@ -575,7 +575,7 @@ fi
     declare -a EESSI_FUSE_MOUNTS=()
    
     # always mount cvmfs-config repo (to get access to software.eessi.io)
    -# EESSI_FUSE_MOUNTS+=("--fusemount" "container:cvmfs2 cvmfs-config.cern.ch /cvmfs/cvmfs-config.cern.ch")
    +EESSI_FUSE_MOUNTS+=("--fusemount" "container:cvmfs2 cvmfs-config.cern.ch /cvmfs/cvmfs-config.cern.ch")
    
     if [[ "${ACCESS}" == "ro" ]]; then
       export EESSI_READONLY="container:cvmfs2 ${repo_name} /cvmfs/${repo_name}"

  • init/Magic_Castle/bash

    diff

    diff --git a/init/Magic_Castle/bash b/init/Magic_Castle/bash
    index 0fb2f67..dbe1772 100644
    --- a/init/Magic_Castle/bash
    +++ b/init/Magic_Castle/bash
    @@ -6,7 +6,7 @@ EESSI_SILENT=1
     source $(dirname "$BASH_SOURCE")/../eessi_environment_variables
    
     # Don't change the default prompt
    -# export PS1="{NESSI $EESSI_VERSION} $ "
    +# export PS1="[EESSI $EESSI_VERSION] $ "
    
     # Provide a clean MODULEPATH
     export MODULEPATH_ROOT=$EESSI_MODULEPATH
    @@ -36,4 +36,4 @@ else
        module reload
     fi
    
    -echo "Environment set up to use NESSI (${EESSI_VERSION}), have fun!"
    +echo "Environment set up to use EESSI (${EESSI_VERSION}), have fun!"

  • init/Magic_Castle/eessi_python3

    diff

    diff --git a/init/Magic_Castle/eessi_python3 b/init/Magic_Castle/eessi_python3
    index c76c7a2..b2f7fd4 100755
    --- a/init/Magic_Castle/eessi_python3
    +++ b/init/Magic_Castle/eessi_python3
    @@ -16,6 +16,6 @@ eessi_python=$(ls ${EESSI_SOFTWARE_PATH}/software/Python/3*GCCcore*/bin/python |
     if [ -f "$eessi_python" ]; then
       $eessi_python "$@"
     else
    -  echo "ERROR: No NESSI Python 3 available."
    +  echo "ERROR: No EESSI Python 3 available."
       false
     fi

  • init/README.md

    diff

    diff --git a/init/README.md b/init/README.md
    index 05baffd..c253dc4 100644
    --- a/init/README.md
    +++ b/init/README.md
    @@ -1,7 +1,7 @@
    -# Bash initialisation for NESSI
    +# Bash initialisation for EESSI
    
     This directory contains the default initialisation script for a bash shell used to
    -configure Lmod and use the NESSI software modules. The (bash)
    +configure Lmod and use the EESSI software modules. The (bash)
     file `eessi_environment_variables` is used to set and export the full set of EESSI
     environment variables:
    
    @@ -9,12 +9,12 @@ environment variables:
     - `EESSI_EPREFIX`: The location of Gentoo Prefix compatability layer (for the architecture).
     - `EESSI_EPREFIX_PYTHON`: Path to `python3` in the Gentoo Prefix layer.
     - `EESSI_SOFTWARE_SUBDIR`: Hardware specific software subdirectory.
    -- `EESSI_SOFTWARE_PATH`: Full path to NESSI software stack.
    +- `EESSI_SOFTWARE_PATH`: Full path to EESSI software stack.
     - `EESSI_MODULEPATH`: Path to be added to the `MODULEPATH`. This can be influenced by two
       externally defined environment varialbes:
         - `EESSI_CUSTOM_MODULEPATH`: defines a fully custom directory to be added to
           `MODULEPATH`, the end user is entirely responsible for what this directory contains.
    -    - `EESSI_MODULE_SUBDIR`: NESSI may ship with a number of possible module naming schemes.
    +    - `EESSI_MODULE_SUBDIR`: EESSI may ship with a number of possible module naming schemes.
           This variable can be used to point to a non-default module naming scheme.
    
     All scripts respect the environment variable `EESSI_SILENT` which, if defined to any
    @@ -25,5 +25,5 @@ value, will make them produce no (non-error) output.
     The `Magic_Castle` subdirectory is home to the bash initialisation that we use for
     [Magic Castle](https://github.com/ComputeCanada/magic_castle).
    
    -It also contains a wrapper for a hardware optimised NESSI Python 3 installation that is used
    +It also contains a wrapper for a hardware optimised EESSI Python 3 installation that is used
     by Magic Castle to properly configure JupyterHub.

  • init/arch_specs/eessi_arch_arm.spec

    diff

    diff --git a/init/arch_specs/eessi_arch_arm.spec b/init/arch_specs/eessi_arch_arm.spec
    index f8f21f9..b5c9275 100755
    --- a/init/arch_specs/eessi_arch_arm.spec
    +++ b/init/arch_specs/eessi_arch_arm.spec
    @@ -1,6 +1,6 @@
     # ARM CPU architecture specifications
     # Software path in EESSI 	| Vendor ID 	| List of defining CPU features
    -"aarch64/neoverse-n1"	"ARM"		"asimd"		# Ampere Altra
    -"aarch64/neoverse-n1"	""		"asimd"		# AWS Graviton2
    -"aarch64/neoverse-v1"	"ARM"		"asimd svei8mm"
    -"aarch64/neoverse-v1"	""		"asimd svei8mm"	# AWS Graviton3
    +"aarch64/neoverse_n1"	"ARM"		"asimd"		# Ampere Altra
    +"aarch64/neoverse_n1"	""		"asimd"		# AWS Graviton2
    +"aarch64/neoverse_v1"	"ARM"		"asimd svei8mm"
    +"aarch64/neoverse_v1"	""		"asimd svei8mm"	# AWS Graviton3

  • init/bash

    diff

    diff --git a/init/bash b/init/bash
    index 26598bb..0029f91 100644
    --- a/init/bash
    +++ b/init/bash
    @@ -5,6 +5,7 @@ function show_msg {
         echo "$msg"
       fi
     }
    +
     # The following method should be safe, but might break if file is a symlink
     # (could switch to $(dirname "$(readlink -f "$BASH_SOURCE")") in that case)
     source $(dirname "$BASH_SOURCE")/eessi_environment_variables
    @@ -12,7 +13,7 @@ source $(dirname "$BASH_SOURCE")/eessi_environment_variables
     # only continue if setting EESSI environment variables worked fine
     if [ $? -eq 0 ]; then
    
    -    export PS1="{NESSI $EESSI_VERSION} $PS1"
    +    export PS1="{EESSI $EESSI_VERSION} $PS1"
    
         # add location of commands provided by compat layer to $PATH;
         # see https://github.com/EESSI/software-layer/issues/52
    @@ -23,7 +24,7 @@ if [ $? -eq 0 ]; then
         source $EESSI_EPREFIX/usr/share/Lmod/init/bash
    
         # prepend location of modules for EESSI software stack to $MODULEPATH
    -    echo "Prepending $EESSI_MODULEPATH to \$MODULEPATH..." >> $output
    +    show_msg "Prepending $EESSI_MODULEPATH to \$MODULEPATH..."
         module use $EESSI_MODULEPATH
    
         #show_msg ""
    @@ -33,6 +34,6 @@ if [ $? -eq 0 ]; then
         #show_msg ""
         #show_msg ""
    
    -    echo "Environment set up to use NESSI (${EESSI_VERSION}), have fun!"
    +    echo "Environment set up to use EESSI (${EESSI_VERSION}), have fun!"
    
     fi

  • init/eessi_archdetect.sh

    diff

    diff --git a/init/eessi_archdetect.sh b/init/eessi_archdetect.sh
    index 236ac5a..58b79b0 100755
    --- a/init/eessi_archdetect.sh
    +++ b/init/eessi_archdetect.sh
    @@ -73,7 +73,7 @@ cpupath(){
         [ $EESSI_SOFTWARE_SUBDIR_OVERRIDE ] && echo ${EESSI_SOFTWARE_SUBDIR_OVERRIDE} && exit
    
         # Identify the best matching CPU architecture from a list of supported specifications for the host CPU
    -    # Return the path to the installation files in NESSI of the best matching architecture
    +    # Return the path to the installation files in EESSI of the best matching architecture
         local cpu_arch_spec=()
    
         # Identify the host CPU architecture
    @@ -118,7 +118,7 @@ cpupath(){
                 # each flag in this CPU specification must be found in the list of flags of the host
                 check_allinfirst "${cpu_flags[*]}" ${arch_spec[2]} && best_arch_match=${arch_spec[0]} && \
                     all_arch_matches="$best_arch_match:$all_arch_matches" && \
    -                log "DEBUG" "cpupath: host CPU best match updated to $best_arch_match"
    +                log "DEBUG" "cpupath: host CPU best match updated to $best_arch_match"
             fi
         done
    

  • init/eessi_defaults

    diff

    diff --git a/init/eessi_defaults b/init/eessi_defaults
    index da057fb..d1779a3 100644
    --- a/init/eessi_defaults
    +++ b/init/eessi_defaults
    @@ -8,9 +8,8 @@
     # license: GPLv2
     #
    
    -export EESSI_CVMFS_REPO="${EESSI_CVMFS_REPO_OVERRIDE:=/cvmfs/pilot.nessi.no}"
    +export EESSI_CVMFS_REPO="${EESSI_CVMFS_REPO_OVERRIDE:=/cvmfs/software.eessi.io}"
     export EESSI_VERSION="${EESSI_VERSION_OVERRIDE:=2023.06}"
     # use archdetect by default, unless otherwise specified
    -# 2024-01-16: switch back to archspec with NESSI for now
    -export EESSI_USE_ARCHDETECT="${EESSI_USE_ARCHDETECT:=0}"
    -export EESSI_USE_ARCHSPEC="${EESSI_USE_ARCHSPEC:=1}"
    +export EESSI_USE_ARCHDETECT="${EESSI_USE_ARCHDETECT:=1}"
    +export EESSI_USE_ARCHSPEC="${EESSI_USE_ARCHSPEC:=0}"

  • init/eessi_environment_variables

    diff

    diff --git a/init/eessi_environment_variables b/init/eessi_environment_variables
    index d4a2e72..3584bfa 100644
    --- a/init/eessi_environment_variables
    +++ b/init/eessi_environment_variables
    @@ -2,6 +2,11 @@
     # $BASH_SOURCE points to correct path, see also http://mywiki.wooledge.org/BashFAQ/028
     EESSI_INIT_DIR_PATH=$(dirname $(realpath $BASH_SOURCE))
    
    +function error() {
    +    echo -e "\e[31mERROR: $1\e[0m" >&2
    +    false
    +}
    +
     function show_msg {
       # only echo msg if EESSI_SILENT is unset
       msg=$1
    @@ -10,16 +15,11 @@ function show_msg {
       fi
     }
    
    -function error() {
    -    echo -e "\e[31mERROR: $1\e[0m" >&2
    -    false
    -}
    -
     # set up minimal environment: $EESSI_PREFIX, $EESSI_VERSION, $EESSI_OS_TYPE, $EESSI_CPU_FAMILY, $EPREFIX
     source $EESSI_INIT_DIR_PATH/minimal_eessi_env
    
     if [ -d $EESSI_PREFIX ]; then
    -  show_msg "Found NESSI repo @ $EESSI_PREFIX!"
    +  show_msg "Found EESSI repo @ $EESSI_PREFIX!"
    
       export EESSI_EPREFIX=$EPREFIX
       if [ -d $EESSI_EPREFIX ]; then
    @@ -63,7 +63,7 @@ if [ -d $EESSI_PREFIX ]; then
                 export EESSI_MODULEPATH=$EESSI_MODULEPATH
                 show_msg "Using ${EESSI_MODULEPATH} as the directory to be added to MODULEPATH."
               else
    -            error "NESSI module path at $EESSI_MODULEPATH not found!"
    +            error "EESSI module path at $EESSI_MODULEPATH not found!"
                 false
               fi
    
    @@ -75,7 +75,7 @@ if [ -d $EESSI_PREFIX ]; then
               fi
    
             else
    -          error "NESSI software layer at $EESSI_SOFTWARE_PATH not found!"
    +          error "EESSI software layer at $EESSI_SOFTWARE_PATH not found!"
             fi
         else
             error "no value set for \$EESSI_SOFTWARE_SUBDIR"
    @@ -84,5 +84,5 @@ if [ -d $EESSI_PREFIX ]; then
         error "Compatibility layer directory $EESSI_EPREFIX not found!"
       fi
     else
    -  error "NESSI repository at $EESSI_PREFIX not found!"
    +  error "EESSI repository at $EESSI_PREFIX not found!"
     fi

  • init/eessi_software_subdir_for_host.py

    diff

    diff --git a/init/eessi_software_subdir_for_host.py b/init/eessi_software_subdir_for_host.py
    index e8f1a71..58e9cfd 100755
    --- a/init/eessi_software_subdir_for_host.py
    +++ b/init/eessi_software_subdir_for_host.py
    @@ -1,6 +1,6 @@
     #!/usr/bin/env python3
     #
    -# Determine NESSI software subdirectory to use for current build host, using archspec
    +# Determine EESSI software subdirectory to use for current build host, using archspec
     #
     import glob
     import os

  • init/minimal_eessi_env

    diff

    diff --git a/init/minimal_eessi_env b/init/minimal_eessi_env
    index f938d1d..5b35620 100644
    --- a/init/minimal_eessi_env
    +++ b/init/minimal_eessi_env
    @@ -1,4 +1,4 @@
    -# define minimal NESSI environment, without relying on external scripts
    +# define minimal EESSI environment, without relying on external scripts
     #
     # this script is *sourced*, not executed, so can't rely on $0 to determine path to self
     # $BASH_SOURCE points to correct path, see also http://mywiki.wooledge.org/BashFAQ/028

• sync scripts and init directories of /cvmfs/pilot.nessi.no with changes in /cvmfs/software.eessi.io

  • CVMFS repo pilot.nessi.no synced with EESSI:
    • removed symlink /cvmfs/pilot.nessi.no/latest
    • added /cvmfs/pilot.nessi.no/README.nessi
    • synced/populated /cvmfs/pilot.nessi.no/versions/2023.06/{init,scripts}
    • should be available soon on CVMFS clients

• check Broadwell supported via archdetect (for building packages)

  • building works without changes
  • using requires an update to a spec file, see add broadwell support to archdetect #248

• check archdetect is used correctly via source /cvmfs/.../init/bash (for using packages)

  • works for Broadwell

  [thomarob@login-2.FRAM ~]$ source /cvmfs/pilot.nessi.no/versions/2023.06/init/bash
  Found NESSI repo @ /cvmfs/pilot.nessi.no/versions/2023.06!
  archdetect says x86_64/intel/broadwell
  Using x86_64/intel/broadwell as software subdirectory.
  Using /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/modules/all as the directory to be added to MODULEPATH.
  Found Lmod configuration file at /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/.lmod/lmodrc.lua
  Initializing Lmod...
  Prepending /cvmfs/pilot.nessi.no/versions/2023.06/software/linux/x86_64/intel/broadwell/modules/all to $MODULEPATH...
  Environment set up to use NESSI (2023.06), have fun!
  

• fix failing CI workflows

[TopRichard]

First complete pass by (except for some easystack files) is done. Remaining todos:

• do second pass by Richard on current differences
  When only differences due to repo naming, repo config, supported CPU architectures --> ✅ is set.

  git diff nessi_2023.06 2023.06-software.eessi.io

  • .github/workflows/test-pilot.nessi.no.yml

    diff

    diff --git a/.github/workflows/test-pilot.nessi.no.yml b/.github/workflows/test-software.eessi.io.yml
    similarity index 76%
    rename from .github/workflows/test-pilot.nessi.no.yml
    rename to .github/workflows/test-software.eessi.io.yml
    index 6342d5d..3ac341a 100644
    --- a/.github/workflows/test-pilot.nessi.no.yml
    +++ b/.github/workflows/test-software.eessi.io.yml
    @@ -1,5 +1,5 @@
     # documentation: https://help.github.com/en/articles/workflow-syntax-for-github-actions
    -name: Check for missing software installations in pilot.nessi.no
    +name: Check for missing software installations in software.eessi.io
     on: [push, pull_request, workflow_dispatch]
     permissions:
       contents: read # to fetch code (actions/checkout)
    @@ -12,37 +12,40 @@ jobs:
             EESSI_VERSION:
             - 2023.06
             EESSI_SOFTWARE_SUBDIR_OVERRIDE:
    -#        - aarch64/generic
    +        - aarch64/generic
    +        - aarch64/neoverse_n1
    +        - aarch64/neoverse_v1
             - x86_64/amd/zen2
    -        - x86_64/intel/broadwell
    +        - x86_64/amd/zen3
    +        - x86_64/intel/haswell
             - x86_64/intel/skylake_avx512
             - x86_64/generic
         steps:
             - name: Check out software-layer repository
               uses: actions/checkout@93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 # v3.1.0
    
    -        - name: Mount NESSI CernVM-FS repository
    +        - name: Mount EESSI CernVM-FS pilot repository
               uses: cvmfs-contrib/github-action-cvmfs@d4641d0d591c9a5c3be23835ced2fb648b44c04b # v3.1
               with:
    -              cvmfs_config_package: https://github.com/NorESSI/filesystem-layer/releases/download/latest/cvmfs-config-nessi_latest_all.deb
    +              cvmfs_config_package: https://github.com/EESSI/filesystem-layer/releases/download/latest/cvmfs-config-eessi_latest_all.deb
                   cvmfs_http_proxy: DIRECT
    -              cvmfs_repositories: pilot.nessi.no
    +              cvmfs_repositories: software.eessi.io
    
             - name: Test check_missing_installations.sh script
               run: |
                   export EESSI_SOFTWARE_SUBDIR_OVERRIDE=${{matrix.EESSI_SOFTWARE_SUBDIR_OVERRIDE}}
    -              source /cvmfs/pilot.nessi.no/versions/${{matrix.EESSI_VERSION}}/init/bash
    +              source /cvmfs/software.eessi.io/versions/${{matrix.EESSI_VERSION}}/init/bash
                   # set $EESSI_CPU_FAMILY to the CPU architecture that corresponds to $EESSI_SOFTWARE_SUBDIR_OVERRIDE (part before the first slash),
                   # to prevent issues with checks in the Easybuild configuration that use this variable
                   export EESSI_CPU_FAMILY=${EESSI_SOFTWARE_SUBDIR_OVERRIDE%%/*}
                   module load EasyBuild
                   which eb
                   eb --version
    -              export EESSI_PREFIX=/cvmfs/pilot.nessi.no/versions/${{matrix.EESSI_VERSION}}
    +              export EESSI_PREFIX=/cvmfs/software.eessi.io/versions/${{matrix.EESSI_VERSION}}
                   export EESSI_OS_TYPE=linux
                   env | grep ^EESSI | sort
    -              echo "just run check_missing_installations.sh (should use easystacks/pilot.nessi.no/${{matrix.EESSI_VERSION}}/eessi-${{matrix.EESSI_VERSION}}-*.yml)"
    -              for easystack_file in $(ls easystacks/pilot.nessi.no/${{matrix.EESSI_VERSION}}/eessi-${{matrix.EESSI_VERSION}}-eb-*.yml); do
    +              echo "just run check_missing_installations.sh (should use easystacks/software.eessi.io/${{matrix.EESSI_VERSION}}/eessi-${{matrix.EESSI_VERSION}}-*.yml)"
    +              for easystack_file in $(ls easystacks/software.eessi.io/${{matrix.EESSI_VERSION}}/eessi-${{matrix.EESSI_VERSION}}-eb-*.yml); do
                       echo "check missing installations for ${easystack_file}..."
                       ./check_missing_installations.sh ${easystack_file}
                       ec=$?
    @@ -52,14 +55,14 @@ jobs:
             - name: Test check_missing_installations.sh with missing package (GCC/8.3.0)
               run: |
                   export EESSI_SOFTWARE_SUBDIR_OVERRIDE=${{matrix.EESSI_SOFTWARE_SUBDIR_OVERRIDE}}
    -              source /cvmfs/pilot.nessi.no/versions/${{matrix.EESSI_VERSION}}/init/bash
    +              source /cvmfs/software.eessi.io/versions/${{matrix.EESSI_VERSION}}/init/bash
                   # set $EESSI_CPU_FAMILY to the CPU architecture that corresponds to $EESSI_SOFTWARE_SUBDIR_OVERRIDE (part before the first slash),
                   # to prevent issues with checks in the Easybuild configuration that use this variable
                   export EESSI_CPU_FAMILY=${EESSI_SOFTWARE_SUBDIR_OVERRIDE%%/*}
                   module load EasyBuild
                   which eb
                   eb --version
    -              export EESSI_PREFIX=/cvmfs/pilot.nessi.no/versions/${{matrix.EESSI_VERSION}}
    +              export EESSI_PREFIX=/cvmfs/software.eessi.io/versions/${{matrix.EESSI_VERSION}}
                   export EESSI_OS_TYPE=linux
                   env | grep ^EESSI | sort
                   # create dummy easystack file with a single entry (something that is not installed in EESSI)

  • .github/workflows/test_eessi_container_script.yml

    diff

    diff --git a/.github/workflows/test_eessi_container_script.yml b/.github/workflows/test_eessi_container_script.yml
    index 3bb67b4..33122e6 100644
    --- a/.github/workflows/test_eessi_container_script.yml
    +++ b/.github/workflows/test_eessi_container_script.yml
    @@ -90,15 +90,15 @@ jobs:
                   elif [[ ${{matrix.SCRIPT_TEST}} == 'readwrite' ]]; then
                     outfile=out_readwrite.txt
                     fn="test_${RANDOM}.txt"
    -                echo "touch /cvmfs/pilot.nessi.no/${fn}" > test_script.sh
    +                echo "touch /cvmfs/software.eessi.io/${fn}" > test_script.sh
                     chmod u+x test_script.sh
                     export SINGULARITY_BIND="$PWD:/test"
                     ./eessi_container.sh --verbose --access rw --mode run /test/test_script.sh > ${outfile}
    
                     tmpdir=$(grep "\-\-resume" ${outfile} | sed "s/.*--resume \([^']*\).*/\1/g")
                     # note: must use '--access rw' again here, since touched file is in overlay upper dir
    -                ./eessi_container.sh --verbose --resume ${tmpdir} --access rw --mode shell <<< "ls -l /cvmfs/pilot.nessi.no/${fn}" > ${outfile}
    -                grep "/cvmfs/pilot.nessi.no/${fn}$" $outfile
    +                ./eessi_container.sh --verbose --resume ${tmpdir} --access rw --mode shell <<< "ls -l /cvmfs/software.eessi.io/${fn}" > ${outfile}
    +                grep "/cvmfs/software.eessi.io/${fn}$" $outfile
    
                   # test use of --resume
                   elif [[ ${{matrix.SCRIPT_TEST}} == 'resume' ]]; then
    @@ -120,12 +120,12 @@ jobs:
                   elif [[ ${{matrix.SCRIPT_TEST}} == 'save' ]]; then
                     outfile=out_save.txt
                     fn="test_${RANDOM}.txt"
    -                test_cmd="touch /cvmfs/pilot.nessi.no/${fn}"
    +                test_cmd="touch /cvmfs/software.eessi.io/${fn}"
                     ./eessi_container.sh --verbose --mode shell --access rw --save test-save.tar <<< "${test_cmd}" 2>&1 | tee ${outfile}
                     rm -f ${outfile}
    
    -                ./eessi_container.sh --verbose --mode shell --access rw --resume test-save.tar <<< "ls -l /cvmfs/pilot.nessi.no/${fn}" > ${outfile}
    -                grep "/cvmfs/pilot.nessi.no/${fn}$" $outfile
    +                ./eessi_container.sh --verbose --mode shell --access rw --resume test-save.tar <<< "ls -l /cvmfs/software.eessi.io/${fn}" > ${outfile}
    +                grep "/cvmfs/software.eessi.io/${fn}$" $outfile
    
                     tar tfv test-save.tar | grep "overlay-upper/${fn}"
    

  • .github/workflows/tests_archdetect.yml

    diff

    diff --git a/.github/workflows/tests_archdetect.yml b/.github/workflows/tests_archdetect.yml
    index 74dbf03..1e8b830 100644
    --- a/.github/workflows/tests_archdetect.yml
    +++ b/.github/workflows/tests_archdetect.yml
    @@ -13,23 +13,23 @@ jobs:
               - x86_64/intel/skylake_avx512/archspec-linux-6132
               - x86_64/amd/zen2/Azure-CentOS7-7V12
               - x86_64/amd/zen3/Azure-CentOS7-7V73X
    -          # commented out since these targets are currently not supported in pilot.nessi.no repo
    +          - aarch64/neoverse_n1/Azure-Ubuntu20-Altra
    +          - aarch64/neoverse_n1/AWS-awslinux-graviton2
    +          - aarch64/neoverse_v1/AWS-awslinux-graviton3
    +          # commented out since these targets are currently not supported in software.eessi.io repo
               # (and some tests assume that the corresponding subdirectory in software layer is there)
               # - ppc64le/power9le/unknown-power9le
    -          # - aarch64/neoverse-n1/Azure-Ubuntu20-Altra
    -          # - aarch64/neoverse-n1/AWS-awslinux-graviton2
    -          # - aarch64/neoverse-v1/AWS-awslinux-graviton3
           fail-fast: false
         steps:
         - name: checkout
           uses: actions/checkout@93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 # v3.1.0
    
    -    - name: Mount NESSI CernVM-FS repository
    +    - name: Mount EESSI CernVM-FS pilot repository
           uses: cvmfs-contrib/github-action-cvmfs@d4641d0d591c9a5c3be23835ced2fb648b44c04b # v3.1
           with:
    -          cvmfs_config_package: https://github.com/NorESSI/filesystem-layer/releases/download/latest/cvmfs-config-nessi_latest_all.deb
    +          cvmfs_config_package: https://github.com/EESSI/filesystem-layer/releases/download/latest/cvmfs-config-eessi_latest_all.deb
               cvmfs_http_proxy: DIRECT
    -          cvmfs_repositories: pilot.nessi.no
    +          cvmfs_repositories: software.eessi.io
    
         - name: test eessi_archdetect.sh
           run: |

  • EESSI-install-software.sh

    diff

    diff --git a/EESSI-install-software.sh b/EESSI-install-software.sh
    index eb616b1..69de9d1 100755
    --- a/EESSI-install-software.sh
    +++ b/EESSI-install-software.sh
    @@ -1,6 +1,6 @@
     #!/bin/bash
     #
    -# Script to install NESSI software stack (version set through init/eessi_defaults)
    +# Script to install EESSI software stack (version set through init/eessi_defaults)
    
     # see example parsing of command line arguments at
     #   https://wiki.bash-hackers.org/scripting/posparams#using_a_while_loop

  • README.md

    diff

    diff --git a/README.md b/README.md
    index 4f06139..ab73ad7 100644
    --- a/README.md
    +++ b/README.md
    @@ -1,25 +1,25 @@
     # Software layer
    
    -The software layer of the NESSI project uses [EasyBuild](https://docs.easybuild.io), [Lmod](https://lmod.readthedocs.io) and [archspec](https://archspec.readthedocs.io).
    +The software layer of the EESSI project uses [EasyBuild](https://docs.easybuild.io), [Lmod](https://lmod.readthedocs.io) and [archspec](https://archspec.readthedocs.io).
    
     See also https://www.eessi.io/docs/software_layer .
    
    -## Accessing the software stack
    +## Pilot software stack
    
     You can set up your environment by sourcing the init script:
    

    -$ source /cvmfs/pilot.nessi.no/versions/2023.06/init/bash
    -Found NESSI repo @ /cvmfs/pilot.nessi.no/versions/2023.06!
    -Derived subdirectory for software layer: x86_64/intel/broadwell
    +$ source /cvmfs/software.eessi.io/versions/2023.06/init/bash
    +Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
    +Derived subdirectory for software layer: x86_64/intel/haswell
    Using x86_64/intel/haswell subdirectory for software layer
    Initializing Lmod...
    -Prepending /cvmfs/pilot.nessi.no/versions/2023.06/software/x86_64/intel/broadwell/modules/all to $MODULEPATH...
    -Environment set up to use NESSI software stack, have fun!
    -{NESSI 2023.06} $
    +Prepending /cvmfs/software.eessi.io/versions/2023.06/software/x86_64/intel/haswell/modules/all to $MODULEPATH...
    +Environment set up to use EESSI (2023.06), have fun!
    +[EESSI 2023.06] $

    
    -### Accessing NESSI via a container
    +### Accessing EESSI via a container
    
    You need Singularity version 3.7 or newer. Then, simply run
    
    @@ -40,6 +40,6 @@ Note, not all features/arguments listed via `./eessi_container.sh --help` are im
    The software in this repository is distributed under the terms of the
    [GNU General Public License v2.0](https://opensource.org/licenses/GPL-2.0).
    
    -See [LICENSE](https://github.com/NorESSI/software-layer/blob/main/LICENSE) for more information.
    +See [LICENSE](https://github.com/EESSI/software-layer/blob/main/LICENSE) for more information.
    
    SPDX-License-Identifier: GPL-2.0-only
    

  • bot/build.sh

    diff

    diff --git a/bot/build.sh b/bot/build.sh
    index 7eb15f3..1622e75 100755
    --- a/bot/build.sh
    +++ b/bot/build.sh
    @@ -128,7 +128,7 @@ EESSI_REPOS_CFG_DIR_OVERRIDE=$(cfg_get_value "repository" "repos_cfg_dir")
     export EESSI_REPOS_CFG_DIR_OVERRIDE=${EESSI_REPOS_CFG_DIR_OVERRIDE:-${PWD}/cfg}
     echo "bot/build.sh: EESSI_REPOS_CFG_DIR_OVERRIDE='${EESSI_REPOS_CFG_DIR_OVERRIDE}'"
    
    -# determine NESSI version to be used from .repository.repo_version in ${JOB_CFG_FILE}
    +# determine EESSI version to be used from .repository.repo_version in ${JOB_CFG_FILE}
     # here, just set & export EESSI_VERSION_OVERRIDE
     # next script (eessi_container.sh) makes use of it via sourcing init scripts
     # (e.g., init/eessi_defaults or init/minimal_eessi_env)

  • build_container.sh 👎

    diff

    diff --git a/build_container.sh b/build_container.sh
    new file mode 100755
    index 0000000..23a9e66
    --- /dev/null
    +++ b/build_container.sh
    @@ -0,0 +1,69 @@
    +#!/bin/bash
    +
    +base_dir=$(dirname $(realpath $0))
    +
    +BUILD_CONTAINER="docker://ghcr.io/eessi/build-node:debian11"
    +
    +if [ $# -lt 2 ]; then
    +    echo "Usage: $0 <shell|run> <path for temporary directories>" >&2
    +    exit 1
    +fi
    +SHELL_OR_RUN=$1
    +EESSI_TMPDIR=$2
    +shift 2
    +
    +if [ "$SHELL_OR_RUN" == "run" ] && [ $# -eq 0 ]; then
    +    echo "ERROR: No command specified to run?!" >&2
    +    exit 1
    +fi
    +
    +# make sure specified temporary directory exists
    +mkdir -p $EESSI_TMPDIR
    +
    +echo "Using $EESSI_TMPDIR as parent for temporary directories..."
    +
    +# create temporary directories
    +mkdir -p $EESSI_TMPDIR/{home,overlay-upper,overlay-work}
    +mkdir -p $EESSI_TMPDIR/{var-lib-cvmfs,var-run-cvmfs}
    +# configure Singularity
    +export SINGULARITY_CACHEDIR=$EESSI_TMPDIR/singularity_cache
    +
    +# take into account that $SINGULARITY_BIND may be defined already, to bind additional paths into the build container
    +BIND_PATHS="$EESSI_TMPDIR/var-run-cvmfs:/var/run/cvmfs,$EESSI_TMPDIR/var-lib-cvmfs:/var/lib/cvmfs,$EESSI_TMPDIR"
    +if [ -z $SINGULARITY_BIND ]; then
    +    export SINGULARITY_BIND="$BIND_PATHS"
    +else
    +    export SINGULARITY_BIND="$SINGULARITY_BIND,$BIND_PATHS"
    +fi
    +
    +# allow that SINGULARITY_HOME is defined before script is run
    +if [ -z $SINGULARITY_HOME ]; then
    +    export SINGULARITY_HOME="$EESSI_TMPDIR/home:/home/$USER"
    +fi
    +
    +source ${base_dir}/init/eessi_defaults
    +# strip "/cvmfs/" from default setting
    +repo_name=${EESSI_CVMFS_REPO/\/cvmfs\//}
    +
    +# set environment variables for fuse mounts in Singularity container
    +export EESSI_PILOT_READONLY="container:cvmfs2 ${repo_name} /cvmfs_ro/${repo_name}"
    +export EESSI_PILOT_WRITABLE_OVERLAY="container:fuse-overlayfs -o lowerdir=/cvmfs_ro/${repo_name} -o upperdir=$EESSI_TMPDIR/overlay-upper -o workdir=$EESSI_TMPDIR/overlay-work ${EESSI_CVMFS_REPO}"
    +
    +# pass $EESSI_SOFTWARE_SUBDIR_OVERRIDE into build container (if set)
    +if [ ! -z ${EESSI_SOFTWARE_SUBDIR_OVERRIDE} ]; then
    +    export SINGULARITYENV_EESSI_SOFTWARE_SUBDIR_OVERRIDE=${EESSI_SOFTWARE_SUBDIR_OVERRIDE}
    +    # also specify via $APPTAINERENV_* (future proof, cfr. https://apptainer.org/docs/user/latest/singularity_compatibility.html#singularity-environment-variable-compatibility)
    +    export APPTAINERENV_EESSI_SOFTWARE_SUBDIR_OVERRIDE=${EESSI_SOFTWARE_SUBDIR_OVERRIDE}
    +fi
    +
    +if [ "$SHELL_OR_RUN" == "shell" ]; then
    +    # start shell in Singularity container, with EESSI repository mounted with writable overlay
    +    echo "Starting Singularity build container..."
    +    singularity shell --fusemount "$EESSI_PILOT_READONLY" --fusemount "$EESSI_PILOT_WRITABLE_OVERLAY" $BUILD_CONTAINER
    +elif [ "$SHELL_OR_RUN" == "run" ]; then
    +    echo "Running '$@' in Singularity build container..."
    +    singularity exec --fusemount "$EESSI_PILOT_READONLY" --fusemount "$EESSI_PILOT_WRITABLE_OVERLAY" $BUILD_CONTAINER "$@"
    +else
    +    echo "ERROR: Unknown action specified: $SHELL_OR_RUN (should be either 'shell' or 'run')" >&2
    +    exit 1
    +fi

    NOTE:The file is removed intentionally by Thomas

  • check_missing_installations.sh

    diff

    diff --git a/check_missing_installations.sh b/check_missing_installations.sh
    index dd292e4..5ea7c5a 100755
    --- a/check_missing_installations.sh
    +++ b/check_missing_installations.sh
    @@ -1,6 +1,6 @@
     #!/bin/bash
     #
    -# Script to check for missing installations in NESSI software stack (version 2023.06)
    +# Script to check for missing installations in EESSI software stack
     #
     # author: Kenneth Hoste (@boegel)
     # author: Thomas Roeblitz (@trz42)

  • configure_easybuild

    diff

    diff --git a/configure_easybuild b/configure_easybuild
    index c1bd1d3..c67b879 100644
    --- a/configure_easybuild
    +++ b/configure_easybuild
    @@ -25,7 +25,6 @@ fi
    
     # note: filtering Bison may break some installations, like Qt5 (see https://github.com/EESSI/software-layer/issues/49)
     # filtering pkg-config breaks R-bundle-Bioconductor installation (see also https://github.com/easybuilders/easybuild-easyconfigs/pull/11104)
    -# problems occur when filtering pkg-config with gnuplot too (picks up Lua 5.1 from $EPREFIX rather than from Lua 5.3 dependency)
     DEPS_TO_FILTER=Autoconf,Automake,Autotools,binutils,bzip2,DBus,flex,gettext,gperf,help2man,intltool,libreadline,libtool,M4,makeinfo,ncurses,util-linux,XZ,zlib
     # For aarch64 we need to also filter out Yasm.
     # See https://github.com/easybuilders/easybuild-easyconfigs/issues/11190

  • create_directory_tarballs.sh

    diff

    diff --git a/create_directory_tarballs.sh b/create_directory_tarballs.sh
    index a1f05b5..0270719 100755
    --- a/create_directory_tarballs.sh
    +++ b/create_directory_tarballs.sh
    @@ -9,7 +9,7 @@ fi
    
     version=$1
    
    -SOFTWARE_LAYER_TARBALL_URL=https://github.com/NorESSI/software-layer/tarball/nessi.no-2023.06
    +SOFTWARE_LAYER_TARBALL_URL="https://github.com/EESSI/software-layer/tarball/${version}-software.eessi.io"
    
     TOPDIR=$(dirname $(realpath $0))
    

  • create_tarball.sh 👎

    diff

    diff --git a/create_tarball.sh b/create_tarball.sh
    index f6239cf..a619df9 100755
    --- a/create_tarball.sh
    +++ b/create_tarball.sh
    @@ -43,7 +43,7 @@ module_files_list=${tmpdir}/module_files.list.txt
     if [ -d ${eessi_version}/software/${os}/${cpu_arch_subdir}/.lmod ]; then
         # include Lmod cache and configuration file (lmodrc.lua),
         # skip whiteout files and backup copies of Lmod cache (spiderT.old.*)
    -    find ${eessi_version}/software/${os}/${cpu_arch_subdir}/.lmod -type f | egrep -v '/\.wh\.|spiderT.old' > ${files_list}
    +    find ${eessi_version}/software/${os}/${cpu_arch_subdir}/.lmod -type f | egrep -v '/\.wh\.|spiderT.old' >> ${files_list}
     fi
    
     # include scripts that were copied by install_scripts.sh, which we want to ship in EESSI repository

    NOTE: Modify > --> >>

  • easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021a.yml

    diff

    diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021a.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021a.yml
    deleted file mode 100644
    index 28612c5..0000000
    --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021a.yml
    +++ /dev/null
    @@ -1,27 +0,0 @@
    -easyconfigs:
    -  - OpenSSL-1.1.eb
    -  # - OpenSSL-1.1.eb:
    -  #     options:
    -  #       include-easyblocks-from-pr: 2922
    -  - git-2.32.0-GCCcore-10.3.0-nodocs.eb
    -  - GCC-10.3.0
    -  - CMake-3.20.1-GCCcore-10.3.0.eb
    -  # - CMake-3.20.1-GCCcore-10.3.0.eb:
    -  #     options:
    -  #       include-easyblocks-from-pr: 2248
    -  - Rust-1.52.1-GCCcore-10.3.0.eb
    -  - foss-2021a.eb
    -  - QuantumESPRESSO-6.7-foss-2021a.eb
    -  - GROMACS-2021.3-foss-2021a.eb:
    -       options:
    -         download-timeout: 1000
    -  - libGLU-9.0.1-GCCcore-10.3.0.eb
    -  # since the BAGEL ec file is new we have to keep the from-pr or eb fails
    -  - BAGEL-1.2.2-foss-2021a.eb:
    -        options:
    -          from-pr: 18446
    -  - WRF-4.3-foss-2021a-dmpar.eb
    -  - Yambo-5.1.1-foss-2021a.eb:
    -        options:
    -          from-pr: 18905
    -

  • easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021b.yml

    diff

    diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021b.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021b.yml
    deleted file mode 100644
    index d838e15..0000000
    --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2021b.yml
    +++ /dev/null
    @@ -1,18 +0,0 @@
    -easyconfigs:
    -  - GCC-11.2.0
    -  - git-2.33.1-GCCcore-11.2.0-nodocs.eb
    -  - CMake-3.21.1-GCCcore-11.2.0.eb
    -  # - CMake-3.21.1-GCCcore-11.2.0.eb:
    -  #     options:
    -  #       include-easyblocks-from-pr: 2248
    -  - OpenMPI-4.1.1-GCC-11.2.0.eb
    -  - FFTW-3.3.10-gompi-2021b.eb
    -  - BLIS-0.8.1-GCC-11.2.0.eb
    -  - foss-2021b.eb
    -  # note, the below CMake has never been installed, just put here to use the
    -  # same version as easy does
    -  # - CMake-3.22.1-GCCcore-11.2.0.eb:
    -  #     options:
    -  #       include-easyblocks-from-pr: 2248
    -  # - zlib-1.2.11-GCCcore-11.2.0
    -  - QuantumESPRESSO-6.8-foss-2021b.eb

  • easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022a.yml

    diff

    diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022a.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022a.yml
    deleted file mode 100644
    index 9991de1..0000000
    --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022a.yml
    +++ /dev/null
    @@ -1,12 +0,0 @@
    -easyconfigs:
    -  - GCC-11.3.0
    -  - git-2.36.0-GCCcore-11.3.0-nodocs.eb
    -  - CMake-3.23.1-GCCcore-11.3.0.eb
    -  # - CMake-3.23.1-GCCcore-11.3.0.eb:
    -  #     options:
    -  #       include-easyblocks-from-pr: 2248
    -  - BLIS-0.9.0-GCC-11.3.0.eb
    -  - OpenMPI-4.1.4-GCC-11.3.0.eb
    -  - FFTW.MPI-3.3.10-gompi-2022a.eb
    -  - foss-2022a.eb
    -  - Python-3.10.4-GCCcore-11.3.0.eb

  • easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022b.yml

    diff

    diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022b.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022b.yml
    deleted file mode 100644
    index 129b709..0000000
    --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-2022b.yml
    +++ /dev/null
    @@ -1,12 +0,0 @@
    -easyconfigs:
    -  - GCC-12.2.0
    -  - git-2.38.1-GCCcore-12.2.0-nodocs.eb
    -  - CMake-3.24.3-GCCcore-12.2.0.eb
    -  # - CMake-3.24.3-GCCcore-12.2.0.eb:
    -  #     options:
    -  #       include-easyblocks-from-pr: 2248
    -  - BLIS-0.9.0-GCC-12.2.0.eb
    -  - OpenMPI-4.1.4-GCC-12.2.0.eb
    -  - FFTW.MPI-3.3.10-gompi-2022b.eb
    -  - Python-3.10.8-GCCcore-12.2.0.eb
    -  - foss-2022b.eb

  • easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-system.yml

    diff

    diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-system.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-system.yml
    deleted file mode 100644
    index a5f84c7..0000000
    --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.7.2-system.yml
    +++ /dev/null
    @@ -1,4 +0,0 @@
    -easyconfigs:
    -  - EasyBuild-4.8.0.eb:
    -      options:
    -        from-pr: 18282

  • easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.0-system.yml

    diff

    diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.0-system.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.0-system.yml
    deleted file mode 100644
    index 96d79de..0000000
    --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.0-system.yml
    +++ /dev/null
    @@ -1,4 +0,0 @@
    -easyconfigs:
    -  - EasyBuild-4.8.1.eb:
    -      options:
    -        from-pr: 18761

  • easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-2022a.yml

    diff

    diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-2022a.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-2022a.yml
    deleted file mode 100644
    index 14f936b..0000000
    --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-2022a.yml
    +++ /dev/null
    @@ -1,66 +0,0 @@
    -easyconfigs:
    -  - BAMM-2.5.0-foss-2022a.eb
    -  - BamTools-2.5.2-GCC-11.3.0.eb
    -  - BCFtools-1.15.1-GCC-11.3.0.eb
    -  - Bowtie2-2.4.5-GCC-11.3.0.eb
    -  - FastTree-2.1.11-GCCcore-11.3.0.eb
    -  - SAMtools-1.16.1-GCC-11.3.0.eb
    -  - gzip-1.12-GCCcore-11.3.0.eb
    -  - lz4-1.9.3-GCCcore-11.3.0.eb
    -  - zstd-1.5.2-GCCcore-11.3.0.eb
    -  - Pillow-9.1.1-GCCcore-11.3.0.eb:
    -      # avoid that hardcoded paths like /usr/include are used in build commands
    -        options:
    -          from-pr: 18881
    -  - Boost.Python-1.79.0-GCC-11.3.0.eb
    -  - netCDF-4.9.0-gompi-2022a.eb:
    -      # use updated CMakeMake easyblock to avoid that -DCMAKE_SKIP_RPATH=ON is used, which breaks the netCDF test step
    -      # see https://github.com/easybuilders/easybuild-easyblocks/pull/3012
    -      options:
    -        include-easyblocks-from-pr: 3012
    -  - NCO-5.1.0-foss-2022a.eb
    -  - AdapterRemoval-2.3.3-GCC-11.3.0.eb
    -  - BEDTools-2.30.0-GCC-11.3.0.eb
    -  - Pysam-0.20.0-GCC-11.3.0.eb
    -  - TransDecoder-5.5.0-GCC-11.3.0.eb
    -  - VCFtools-0.1.16-GCC-11.3.0.eb
    -  - XML-LibXML-2.0207-GCCcore-11.3.0.eb
    -  - CMake-3.24.3-GCCcore-11.3.0.eb
    -  - elfutils-0.187-GCCcore-11.3.0.eb
    -  - Ninja-1.10.2-GCCcore-11.3.0.eb
    -  - Z3-4.10.2-GCCcore-11.3.0.eb
    -  - SciPy-bundle-2022.05-foss-2022a.eb
    -  - Xvfb-21.1.3-GCCcore-11.3.0.eb:
    -      # enable exec permissions for xvfb-run after copying;
    -      # need to also enable user write permissions on xvfb-run to ensure that copying with preserved permissions works
    -      options:
    -        from-pr: 18834
    -  - tbb-2021.5.0-GCCcore-11.3.0.eb
    -  - CMSeq-1.0.4-foss-2022a.eb
    -  - Pango-1.50.7-GCCcore-11.3.0.eb
    -  - nodejs-16.15.1-GCCcore-11.3.0.eb:
    -      options:
    -        download-timeout: 1000
    -  - ecCodes-2.27.0-gompi-2022a.eb:
    -        options:
    -          download-timeout: 1000
    -  - GLPK-5.0-GCCcore-11.3.0.eb
    -  - libgit2-1.4.3-GCCcore-11.3.0.eb
    -  - libGLU-9.0.2-GCCcore-11.3.0.eb
    -  - libsndfile-1.1.0-GCCcore-11.3.0.eb
    -  - LibTIFF-4.3.0-GCCcore-11.3.0.eb
    -  - MPFR-4.1.0-GCCcore-11.3.0.eb
    -  - NLopt-2.7.1-GCCcore-11.3.0.eb
    -  - PCRE2-10.40-GCCcore-11.3.0.eb
    -  - Tk-8.6.12-GCCcore-11.3.0.eb
    -  - GROMACS-2023.1-foss-2022a.eb
    -  - MUMPS-5.5.1-foss-2022a-metis.eb
    -  - Pillow-9.1.1-GCCcore-11.3.0.eb:
    -      # Uses a custom hook since has zlib as dependency which has hard coded header and library path within Pillow code.
    -      options:
    -        from-pr: 18881
    -  - ParaView-5.10.1-foss-2022a-mpi.eb:
    -        options:
    -          download-timeout: 1000
    -  - ASE-3.22.1-foss-2022a.eb
    -  - HISAT2-2.2.1-gompi-2022a.eb

  • easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.2-2022a.yml

    diff

    diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.2-2022a.yml b/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.2-2022a.yml
    deleted file mode 100644
    index f191e76..0000000
    --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.2-2022a.yml
    +++ /dev/null
    @@ -1,3 +0,0 @@
    -easyconfigs:
    -  - CDO-2.0.6-gompi-2022a.eb
    -  - OpenBabel-3.1.1-gompi-2022a.eb

  • easystacks/pilot.nessi.no/2023.06/README.md

    diff

    diff --git a/easystacks/pilot.nessi.no/2023.06/README.md b/easystacks/software.eessi.io/2023.06/README.md
    similarity index 100%
    rename from easystacks/pilot.nessi.no/2023.06/README.md
    rename to easystacks/software.eessi.io/2023.06/README.md

  • easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-system.yml

    diff

    diff --git a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-system.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-001-system.yml
    similarity index 54%
    rename from easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-system.yml
    rename to easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-001-system.yml
    index 7418c0a..f02b9f2 100644
    --- a/easystacks/pilot.nessi.no/2023.06/eessi-2023.06-eb-4.8.1-system.yml
    +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-001-system.yml
    @@ -1,6 +1,7 @@
     easyconfigs:
    -  # wraps around Java/11.0.20
    -  - Java-11.eb
       - EasyBuild-4.8.2.eb:
           options:
             from-pr: 19105
    +  - Nextflow-23.10.0.eb:
    +      options:
    +        from-pr: 19172

  • easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-010-CUDA.yml

    diff

    diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-010-CUDA.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-010-CUDA.yml
    new file mode 100644
    index 0000000..dda274b
    --- /dev/null
    +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-010-CUDA.yml
    @@ -0,0 +1,5 @@
    +easyconfigs:
    +  - CUDA-12.1.1.eb:
    +      options:
    +        include-easyblocks-from-pr: 3045
    +        accept-eula-for: CUDA

  • easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2022b.yml

    diff

    diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2022b.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2022b.yml
    new file mode 100644
    index 0000000..69ffb75
    --- /dev/null
    +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2022b.yml
    @@ -0,0 +1,6 @@
    +easyconfigs:
    +  - foss-2022b.eb
    +  - HarfBuzz-5.3.1-GCCcore-12.2.0.eb:
    +      options:
    +        from-pr: 19339
    +  - Qt5-5.15.7-GCCcore-12.2.0.eb

  • easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2023a.yml

    diff

    diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2023a.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2023a.yml
    new file mode 100644
    index 0000000..aab910d
    --- /dev/null
    +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.8.2-2023a.yml
    @@ -0,0 +1,48 @@
    +easyconfigs:
    +  - GCC-12.3.0.eb
    +  - Rust-1.70.0-GCCcore-12.3.0.eb:
    +      # fix build of Rust 1.70.0 by disabling download of pre-built LLVM;
    +      # see https://github.com/easybuilders/easybuild-easyblocks/pull/3038
    +      options:
    +        include-easyblocks-from-pr: 3038
    +  - foss-2023a.eb
    +  - pybind11-2.11.1-GCCcore-12.3.0.eb:
    +      # avoid indirect dependency on old CMake version built with GCCcore/10.2.0 via Catch2 build dependency;
    +      # see https://github.com/easybuilders/easybuild-easyconfigs/pull/19270
    +      options:
    +        from-pr: 19270
    +  - SciPy-bundle-2023.07-gfbf-2023a.eb
    +  - TensorFlow-2.13.0-foss-2023a.eb:
    +      # patch setup.py for grpcio extension in TensorFlow 2.13.0 easyconfigs to take into account alternate sysroot;
    +      # see https://github.com/easybuilders/easybuild-easyconfigs/pull/19268
    +      options:
    +        from-pr: 19268
    +  - X11-20230603-GCCcore-12.3.0.eb
    +  - HarfBuzz-5.3.1-GCCcore-12.3.0.eb:
    +      options:
    +        from-pr: 19339
    +  - Qt5-5.15.10-GCCcore-12.3.0.eb
    +  - OSU-Micro-Benchmarks-7.1-1-gompi-2023a.eb
    +  - LHAPDF-6.5.4-GCC-12.3.0.eb:
    +      options:
    +        from-pr: 19363
    +  - LoopTools-2.15-GCC-12.3.0.eb:
    +      options:
    +        from-pr: 19397
    +  - R-4.3.2-gfbf-2023a.eb:
    +      options:
    +        from-pr: 19185
    +  - Boost-1.82.0-GCC-12.3.0.eb
    +  - netCDF-4.9.2-gompi-2023a.eb
    +  - FFmpeg-6.0-GCCcore-12.3.0.eb
    +  - CUDA-Samples-12.1-GCC-12.3.0-CUDA-12.1.1.eb:
    +      # use easyconfig that only install subset of CUDA samples,
    +      # to circumvent problem with nvcc linking to glibc of host OS,
    +      # see https://github.com/easybuilders/easybuild-easyconfigs/pull/19189;
    +      # and where additional samples are excluded because they fail to build on aarch64,
    +      # see https://github.com/easybuilders/easybuild-easyconfigs/pull/19451;
    +      options:
    +        from-pr: 19451
    +  - ALL-0.9.2-foss-2023a.eb:
    +      options:
    +        from-pr: 19455

  • easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-001-system.yml

    diff

    diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-001-system.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-001-system.yml
    new file mode 100644
    index 0000000..25c13e4
    --- /dev/null
    +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-001-system.yml
    @@ -0,0 +1,5 @@
    +easyconfigs:
    +  - EasyBuild-4.9.0.eb:
    +      options:
    +        from-pr: 19464
    +  - ReFrame-4.3.3.eb

  • easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2022b.yml

    diff

    diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2022b.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2022b.yml
    new file mode 100644
    index 0000000..fdacd95
    --- /dev/null
    +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2022b.yml
    @@ -0,0 +1,2 @@
    +easyconfigs:
    +  - SciPy-bundle-2023.02-gfbf-2022b.eb

  • easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023a.yml

    diff

    diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023a.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023a.yml
    new file mode 100644
    index 0000000..eba776a
    --- /dev/null
    +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023a.yml
    @@ -0,0 +1,10 @@
    +easyconfigs:
    +  - OpenFOAM-11-foss-2023a.eb:
    +      # see https://github.com/easybuilders/easybuild-easyconfigs/pull/19545
    +      options:
    +        from-pr: 19545
    +  - at-spi2-core-2.49.91-GCCcore-12.3.0.eb
    +  - ESPResSo-4.2.1-foss-2023a.eb:
    +      # see https://github.com/easybuilders/easybuild-easyconfigs/pull/19592
    +      options:
    +        from-pr: 19592

  • easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023b.yml

    diff

    diff --git a/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023b.yml b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023b.yml
    new file mode 100644
    index 0000000..9a3b8ef
    --- /dev/null
    +++ b/easystacks/software.eessi.io/2023.06/eessi-2023.06-eb-4.9.0-2023b.yml
    @@ -0,0 +1,7 @@
    +easyconfigs:
    +  - GCC-13.2.0.eb
    +  - foss-2023b.eb
    +  - SciPy-bundle-2023.11-gfbf-2023b.eb
    +  - netCDF-4.9.2-gompi-2023b.eb:
    +      options:
    +        from-pr: 19534

  • eb_hooks.py 👎

    diff

    diff --git a/eb_hooks.py b/eb_hooks.py
    index b3cbc4b..2bc2fde 100644
    --- a/eb_hooks.py
    +++ b/eb_hooks.py
    @@ -185,20 +185,21 @@ def parse_hook_fontconfig_add_fonts(ec, eprefix):
    
    
     def parse_hook_openblas_relax_lapack_tests_num_errors(ec, eprefix):
    -    """Relax number of failing numerical LAPACK tests for aarch64/neoverse_v1 CPU target."""
    +    """Relax number of failing numerical LAPACK tests for aarch64/neoverse_v1 CPU target for OpenBLAS < 0.3.23"""
         cpu_target = get_eessi_envvar('EESSI_SOFTWARE_SUBDIR')
         if ec.name == 'OpenBLAS':
    -        # relax maximum number of failed numerical LAPACK tests for aarch64/neoverse_v1 CPU target
    -        # since the default setting of 150 that works well on other aarch64 targets and x86_64 is a bit too strict
    -        # See https://github.com/EESSI/software-layer/issues/314
    -        cfg_option = 'max_failing_lapack_tests_num_errors'
    -        if cpu_target == CPU_TARGET_NEOVERSE_V1:
    -            orig_value = ec[cfg_option]
    -            ec[cfg_option] = 400
    -            print_msg("Maximum number of failing LAPACK tests with numerical errors for %s relaxed to %s (was %s)",
    -                      ec.name, ec[cfg_option], orig_value)
    -        else:
    -            print_msg("Not changing option %s for %s on non-AARCH64", cfg_option, ec.name)
    +        if LooseVersion(ec.version) < LooseVersion('0.3.23'):
    +            # relax maximum number of failed numerical LAPACK tests for aarch64/neoverse_v1 CPU target
    +            # since the default setting of 150 that works well on other aarch64 targets and x86_64 is a bit too strict
    +            # See https://github.com/EESSI/software-layer/issues/314
    +            cfg_option = 'max_failing_lapack_tests_num_errors'
    +            if cpu_target == CPU_TARGET_NEOVERSE_V1:
    +                orig_value = ec[cfg_option]
    +                ec[cfg_option] = 400
    +                print_msg("Maximum number of failing LAPACK tests with numerical errors for %s relaxed to %s (was %s)",
    +                          ec.name, ec[cfg_option], orig_value)
    +            else:
    +                print_msg("Not changing option %s for %s on non-AARCH64", cfg_option, ec.name)
         else:
             raise EasyBuildError("OpenBLAS-specific hook triggered for non-OpenBLAS easyconfig?!")
    
    @@ -393,6 +394,18 @@ def pre_test_hook_ignore_failing_tests_SciPybundle(self, *args, **kwargs):
         if self.name == 'SciPy-bundle' and self.version in scipy_bundle_versions and cpu_target == CPU_TARGET_NEOVERSE_V1:
             self.cfg['testopts'] = "|| echo ignoring failing tests"
    
    +def pre_test_hook_ignore_failing_tests_netCDF(self, *args, **kwargs):
    +    """
    +    Pre-test hook for netCDF: skip failing tests for selected netCDF versions on neoverse_v1
    +    cfr. https://github.com/EESSI/software-layer/issues/425
    +    The following tests are problematic:
    +        163 - nc_test4_run_par_test (Timeout)
    +        190 - h5_test_run_par_tests (Timeout)
    +    A few other tests are skipped in the easyconfig and patches for similar issues, see above issue for details.
    +    """
    +    cpu_target = get_eessi_envvar('EESSI_SOFTWARE_SUBDIR')
    +    if self.name == 'netCDF' and self.version == '4.9.2' and cpu_target == CPU_TARGET_NEOVERSE_V1:
    +        self.cfg['testopts'] = "|| echo ignoring failing tests"
    
     def pre_single_extension_hook(ext, *args, **kwargs):
         """Main pre-extension: trigger custom functions based on software name."""
    @@ -573,6 +586,7 @@ PRE_TEST_HOOKS = {
         'ESPResSo': pre_test_hook_ignore_failing_tests_ESPResSo,
         'FFTW.MPI': pre_test_hook_ignore_failing_tests_FFTWMPI,
         'SciPy-bundle': pre_test_hook_ignore_failing_tests_SciPybundle,
    +    'netCDF': pre_test_hook_ignore_failing_tests_netCDF,
     }
    
     PRE_SINGLE_EXTENSION_HOOKS = {

    NOTE:Newly added changes

  • eessi-2023.06-known-issues.yml 👎

    diff

    diff --git a/eessi-2023.06-known-issues.yml b/eessi-2023.06-known-issues.yml
    new file mode 100644
    index 0000000..475ee2c
    --- /dev/null
    +++ b/eessi-2023.06-known-issues.yml
    @@ -0,0 +1,28 @@
    +- aarch64/neoverse_v1:
    +  - ESPResSo-4.2.1-foss-2023a:
    +      - issue: https://github.com/EESSI/software-layer/issues/363
    +      - info: "ESPResSo tests failing due to timeouts"
    +  - FFTW.MPI-3.3.10-gompi-2023a:
    +      - issue: https://github.com/EESSI/software-layer/issues/325
    +      - info: "Flaky FFTW tests, random failures"
    +  - FFTW.MPI-3.3.10-gompi-2023b:
    +      - issue: https://github.com/EESSI/software-layer/issues/325
    +      - info: "Flaky FFTW tests, random failures"
    +  - netCDF-4.9.2-gompi-2023a.eb:
    +      - issue: https://github.com/EESSI/software-layer/issues/425
    +      - info: "netCDF intermittent test failures"
    +  - netCDF-4.9.2-gompi-2023b.eb:
    +      - issue: https://github.com/EESSI/software-layer/issues/425
    +      - info: "netCDF intermittent test failures"
    +  - OpenBLAS-0.3.21-GCC-12.2.0:
    +    - issue: https://github.com/EESSI/software-layer/issues/314
    +    - info: "Increased number of numerical errors in OpenBLAS test suite (344 vs max. 150 on x86_64/*)"
    +  - SciPy-bundle-2023.02-gfbf-2022b:
    +    - issue: https://github.com/EESSI/software-layer/issues/318
    +    - info: "numpy built with -march=armv8.4-a instead of -mcpu=native (no SVE) + 2 failing tests (vs 50005 passed) in scipy test suite"
    +  - SciPy-bundle-2023.07-gfbf-2023a:
    +    - issue: https://github.com/EESSI/software-layer/issues/318
    +    - info: "2 failing tests (vs 54409 passed) in scipy test suite"
    +  - SciPy-bundle-2023.11-gfbf-2023b:
    +    - issue: https://github.com/EESSI/software-layer/issues/318
    +    - info: "2 failing tests (vs 54876 passed) in scipy test suite"

    NOTE: Newly added changes

  • eessi_container.sh 👎

    diff

    diff --git a/eessi_container.sh b/eessi_container.sh
    index bfc5963..d6e9558 100755
    --- a/eessi_container.sh
    +++ b/eessi_container.sh
    @@ -30,8 +30,8 @@
     # -. initial settings & exit codes
     TOPDIR=$(dirname $(realpath $0))
    
    -source ${TOPDIR}/scripts/utils.sh
    -source ${TOPDIR}/scripts/cfg_files.sh
    +source "${TOPDIR}"/scripts/utils.sh
    +source "${TOPDIR}"/scripts/cfg_files.sh
    
     # exit codes: bitwise shift codes to allow for combination of exit codes
     # ANY_ERROR_EXITCODE is sourced from ${TOPDIR}/scripts/utils.sh
    @@ -83,7 +83,7 @@ display_help() {
       echo "                           MODE==run (run a script or command) [default: shell]"
       echo "  -n | --nvidia MODE     - configure the container to work with NVIDIA GPUs,"
       echo "                           MODE==install for a CUDA installation, MODE==run to"
    -  echo "                           attach a GPU, MODE==all for both [default: false]"
    +  echo "                           attach a GPU, MODE==all for both [default: false]"
       echo "  -r | --repository CFG  - configuration file or identifier defining the"
       echo "                           repository to use [default: EESSI via"
       echo "                           default container, see --container]"
    @@ -164,7 +164,7 @@ while [[ $# -gt 0 ]]; do
           SETUP_NVIDIA=1
           NVIDIA_MODE="$2"
           shift 2
    -      ;;
    +      ;;
         -r|--repository)
           REPOSITORY="$2"
           shift 2
    @@ -575,7 +575,7 @@ fi
     declare -a EESSI_FUSE_MOUNTS=()
    
     # always mount cvmfs-config repo (to get access to software.eessi.io)
    -# EESSI_FUSE_MOUNTS+=("--fusemount" "container:cvmfs2 cvmfs-config.cern.ch /cvmfs/cvmfs-config.cern.ch")
    +EESSI_FUSE_MOUNTS+=("--fusemount" "container:cvmfs2 cvmfs-config.cern.ch /cvmfs/cvmfs-config.cern.ch")
    
     if [[ "${ACCESS}" == "ro" ]]; then
       export EESSI_READONLY="container:cvmfs2 ${repo_name} /cvmfs/${repo_name}"

    NOTE: Add quotes + commented out intentionally

  • init/Magic_Castle/bash

    diff

    diff --git a/init/Magic_Castle/bash b/init/Magic_Castle/bash
    index 0fb2f67..dbe1772 100644
    --- a/init/Magic_Castle/bash
    +++ b/init/Magic_Castle/bash
    @@ -6,7 +6,7 @@ EESSI_SILENT=1
     source $(dirname "$BASH_SOURCE")/../eessi_environment_variables
    
     # Don't change the default prompt
    -# export PS1="{NESSI $EESSI_VERSION} $ "
    +# export PS1="[EESSI $EESSI_VERSION] $ "
    
     # Provide a clean MODULEPATH
     export MODULEPATH_ROOT=$EESSI_MODULEPATH
    @@ -36,4 +36,4 @@ else
        module reload
     fi
    
    -echo "Environment set up to use NESSI (${EESSI_VERSION}), have fun!"
    +echo "Environment set up to use EESSI (${EESSI_VERSION}), have fun!"

  • init/Magic_Castle/eessi_python3

    diff

    diff --git a/init/Magic_Castle/eessi_python3 b/init/Magic_Castle/eessi_python3
    index c76c7a2..b2f7fd4 100755
    --- a/init/Magic_Castle/eessi_python3
    +++ b/init/Magic_Castle/eessi_python3
    @@ -16,6 +16,6 @@ eessi_python=$(ls ${EESSI_SOFTWARE_PATH}/software/Python/3*GCCcore*/bin/python |
     if [ -f "$eessi_python" ]; then
       $eessi_python "$@"
     else
    -  echo "ERROR: No NESSI Python 3 available."
    +  echo "ERROR: No EESSI Python 3 available."
       false
     fi

  • init/README.md

    diff

    diff --git a/init/README.md b/init/README.md
    index 05baffd..c253dc4 100644
    --- a/init/README.md
    +++ b/init/README.md
    @@ -1,7 +1,7 @@
    -# Bash initialisation for NESSI
    +# Bash initialisation for EESSI
    
     This directory contains the default initialisation script for a bash shell used to
    -configure Lmod and use the NESSI software modules. The (bash)
    +configure Lmod and use the EESSI software modules. The (bash)
     file `eessi_environment_variables` is used to set and export the full set of EESSI
     environment variables:
    
    @@ -9,12 +9,12 @@ environment variables:
     - `EESSI_EPREFIX`: The location of Gentoo Prefix compatability layer (for the architecture).
     - `EESSI_EPREFIX_PYTHON`: Path to `python3` in the Gentoo Prefix layer.
     - `EESSI_SOFTWARE_SUBDIR`: Hardware specific software subdirectory.
    -- `EESSI_SOFTWARE_PATH`: Full path to NESSI software stack.
    +- `EESSI_SOFTWARE_PATH`: Full path to EESSI software stack.
     - `EESSI_MODULEPATH`: Path to be added to the `MODULEPATH`. This can be influenced by two
       externally defined environment varialbes:
         - `EESSI_CUSTOM_MODULEPATH`: defines a fully custom directory to be added to
           `MODULEPATH`, the end user is entirely responsible for what this directory contains.
    -    - `EESSI_MODULE_SUBDIR`: NESSI may ship with a number of possible module naming schemes.
    +    - `EESSI_MODULE_SUBDIR`: EESSI may ship with a number of possible module naming schemes.
           This variable can be used to point to a non-default module naming scheme.
    
     All scripts respect the environment variable `EESSI_SILENT` which, if defined to any
    @@ -25,5 +25,5 @@ value, will make them produce no (non-error) output.
     The `Magic_Castle` subdirectory is home to the bash initialisation that we use for
     [Magic Castle](https://github.com/ComputeCanada/magic_castle).
    
    -It also contains a wrapper for a hardware optimised NESSI Python 3 installation that is used
    +It also contains a wrapper for a hardware optimised EESSI Python 3 installation that is used
     by Magic Castle to properly configure JupyterHub.

  • init/arch_specs/eessi_arch_arm.spec

    diff

    diff --git a/init/arch_specs/eessi_arch_arm.spec b/init/arch_specs/eessi_arch_arm.spec
    index f8f21f9..b5c9275 100755
    --- a/init/arch_specs/eessi_arch_arm.spec
    +++ b/init/arch_specs/eessi_arch_arm.spec
    @@ -1,6 +1,6 @@
     # ARM CPU architecture specifications
     # Software path in EESSI 	| Vendor ID 	| List of defining CPU features
    -"aarch64/neoverse-n1"	"ARM"		"asimd"		# Ampere Altra
    -"aarch64/neoverse-n1"	""		"asimd"		# AWS Graviton2
    -"aarch64/neoverse-v1"	"ARM"		"asimd svei8mm"
    -"aarch64/neoverse-v1"	""		"asimd svei8mm"	# AWS Graviton3
    +"aarch64/neoverse_n1"	"ARM"		"asimd"		# Ampere Altra
    +"aarch64/neoverse_n1"	""		"asimd"		# AWS Graviton2
    +"aarch64/neoverse_v1"	"ARM"		"asimd svei8mm"
    +"aarch64/neoverse_v1"	""		"asimd svei8mm"	# AWS Graviton3

  • init/bash 👎

    diff

    diff --git a/init/bash b/init/bash
    index 26598bb..0029f91 100644
    --- a/init/bash
    +++ b/init/bash
    @@ -5,6 +5,7 @@ function show_msg {
         echo "$msg"
       fi
     }
    +
     # The following method should be safe, but might break if file is a symlink
     # (could switch to $(dirname "$(readlink -f "$BASH_SOURCE")") in that case)
     source $(dirname "$BASH_SOURCE")/eessi_environment_variables
    @@ -12,7 +13,7 @@ source $(dirname "$BASH_SOURCE")/eessi_environment_variables
     # only continue if setting EESSI environment variables worked fine
     if [ $? -eq 0 ]; then
    
    -    export PS1="{NESSI $EESSI_VERSION} $PS1"
    +    export PS1="{EESSI $EESSI_VERSION} $PS1"
    
         # add location of commands provided by compat layer to $PATH;
         # see https://github.com/EESSI/software-layer/issues/52
    @@ -23,7 +24,7 @@ if [ $? -eq 0 ]; then
         source $EESSI_EPREFIX/usr/share/Lmod/init/bash
    
         # prepend location of modules for EESSI software stack to $MODULEPATH
    -    echo "Prepending $EESSI_MODULEPATH to \$MODULEPATH..." >> $output
    +    show_msg "Prepending $EESSI_MODULEPATH to \$MODULEPATH..."
         module use $EESSI_MODULEPATH
    
         #show_msg ""
    @@ -33,6 +34,6 @@ if [ $? -eq 0 ]; then
         #show_msg ""
         #show_msg ""
    
    -    echo "Environment set up to use NESSI (${EESSI_VERSION}), have fun!"
    +    echo "Environment set up to use EESSI (${EESSI_VERSION}), have fun!"
    
     fi

    NOTE: need to change echo -->show_msg.

  • init/eessi_archdetect.sh 👎

    diff

    diff --git a/init/eessi_archdetect.sh b/init/eessi_archdetect.sh
    index 236ac5a..58b79b0 100755
    --- a/init/eessi_archdetect.sh
    +++ b/init/eessi_archdetect.sh
    @@ -73,7 +73,7 @@ cpupath(){
         [ $EESSI_SOFTWARE_SUBDIR_OVERRIDE ] && echo ${EESSI_SOFTWARE_SUBDIR_OVERRIDE} && exit
    
         # Identify the best matching CPU architecture from a list of supported specifications for the host CPU
    -    # Return the path to the installation files in NESSI of the best matching architecture
    +    # Return the path to the installation files in EESSI of the best matching architecture
         local cpu_arch_spec=()
    
         # Identify the host CPU architecture
    @@ -118,7 +118,7 @@ cpupath(){
                 # each flag in this CPU specification must be found in the list of flags of the host
                 check_allinfirst "${cpu_flags[*]}" ${arch_spec[2]} && best_arch_match=${arch_spec[0]} && \
                     all_arch_matches="$best_arch_match:$all_arch_matches" && \
    -                log "DEBUG" "cpupath: host CPU best match updated to $best_arch_match"
    +                log "DEBUG" "cpupath: host CPU best match updated to $best_arch_match"
             fi
         done
    

    NOTE: Extra space character in NESSI.

  • init/eessi_defaults

    diff

    diff --git a/init/eessi_defaults b/init/eessi_defaults
    index da057fb..d1779a3 100644
    --- a/init/eessi_defaults
    +++ b/init/eessi_defaults
    @@ -8,9 +8,8 @@
     # license: GPLv2
     #
    
    -export EESSI_CVMFS_REPO="${EESSI_CVMFS_REPO_OVERRIDE:=/cvmfs/pilot.nessi.no}"
    +export EESSI_CVMFS_REPO="${EESSI_CVMFS_REPO_OVERRIDE:=/cvmfs/software.eessi.io}"
     export EESSI_VERSION="${EESSI_VERSION_OVERRIDE:=2023.06}"
     # use archdetect by default, unless otherwise specified
    -# 2024-01-16: switch back to archspec with NESSI for now
    -export EESSI_USE_ARCHDETECT="${EESSI_USE_ARCHDETECT:=0}"
    -export EESSI_USE_ARCHSPEC="${EESSI_USE_ARCHSPEC:=1}"
    +export EESSI_USE_ARCHDETECT="${EESSI_USE_ARCHDETECT:=1}"
    +export EESSI_USE_ARCHSPEC="${EESSI_USE_ARCHSPEC:=0}"

  • init/eessi_environment_variables 👎

    diff

    diff --git a/init/eessi_environment_variables b/init/eessi_environment_variables
    index d4a2e72..3584bfa 100644
    --- a/init/eessi_environment_variables
    +++ b/init/eessi_environment_variables
    @@ -2,6 +2,11 @@
     # $BASH_SOURCE points to correct path, see also http://mywiki.wooledge.org/BashFAQ/028
     EESSI_INIT_DIR_PATH=$(dirname $(realpath $BASH_SOURCE))
    
    +function error() {
    +    echo -e "\e[31mERROR: $1\e[0m" >&2
    +    false
    +}
    +
     function show_msg {
       # only echo msg if EESSI_SILENT is unset
       msg=$1
    @@ -10,16 +15,11 @@ function show_msg {
       fi
     }
    
    -function error() {
    -    echo -e "\e[31mERROR: $1\e[0m" >&2
    -    false
    -}
    -
     # set up minimal environment: $EESSI_PREFIX, $EESSI_VERSION, $EESSI_OS_TYPE, $EESSI_CPU_FAMILY, $EPREFIX
     source $EESSI_INIT_DIR_PATH/minimal_eessi_env
    
     if [ -d $EESSI_PREFIX ]; then
    -  show_msg "Found NESSI repo @ $EESSI_PREFIX!"
    +  show_msg "Found EESSI repo @ $EESSI_PREFIX!"
    
       export EESSI_EPREFIX=$EPREFIX
       if [ -d $EESSI_EPREFIX ]; then
    @@ -63,7 +63,7 @@ if [ -d $EESSI_PREFIX ]; then
                 export EESSI_MODULEPATH=$EESSI_MODULEPATH
                 show_msg "Using ${EESSI_MODULEPATH} as the directory to be added to MODULEPATH."
               else
    -            error "NESSI module path at $EESSI_MODULEPATH not found!"
    +            error "EESSI module path at $EESSI_MODULEPATH not found!"
                 false
               fi
    
    @@ -75,7 +75,7 @@ if [ -d $EESSI_PREFIX ]; then
               fi
    
             else
    -          error "NESSI software layer at $EESSI_SOFTWARE_PATH not found!"
    +          error "EESSI software layer at $EESSI_SOFTWARE_PATH not found!"
             fi
         else
             error "no value set for \$EESSI_SOFTWARE_SUBDIR"
    @@ -84,5 +84,5 @@ if [ -d $EESSI_PREFIX ]; then
         error "Compatibility layer directory $EESSI_EPREFIX not found!"
       fi
     else
    -  error "NESSI repository at $EESSI_PREFIX not found!"
    +  error "EESSI repository at $EESSI_PREFIX not found!"
     fi

    NOTE: code line reordering .

  • init/eessi_software_subdir_for_host.py

    diff

    diff --git a/init/eessi_software_subdir_for_host.py b/init/eessi_software_subdir_for_host.py
    index e8f1a71..58e9cfd 100755
    --- a/init/eessi_software_subdir_for_host.py
    +++ b/init/eessi_software_subdir_for_host.py
    @@ -1,6 +1,6 @@
     #!/usr/bin/env python3
     #
    -# Determine NESSI software subdirectory to use for current build host, using archspec
    +# Determine EESSI software subdirectory to use for current build host, using archspec
     #
     import glob
     import os

  • init/minimal_eessi_env

    diff

    diff --git a/init/minimal_eessi_env b/init/minimal_eessi_env
    index f938d1d..5b35620 100644
    --- a/init/minimal_eessi_env
    +++ b/init/minimal_eessi_env
    @@ -1,4 +1,4 @@
    -# define minimal NESSI environment, without relying on external scripts
    +# define minimal EESSI environment, without relying on external scripts
     #
     # this script is *sourced*, not executed, so can't rely on $0 to determine path to self
     # $BASH_SOURCE points to correct path, see also http://mywiki.wooledge.org/BashFAQ/028