Syncing code-base and CVMFS repository with EESSI (April 2024)

[TopRichard]

In January we synced the NESSI code-base with EESSI (see #223 and https://github.com/NorESSI/software-layer/issues?q=label%3Async-with-EESSI+is%3Aclosed). The last PR in that effort was #256 (merged Jan 28, 10:08 PM). Since then 49 PRs have been merged in EESSI, see https://github.com/EESSI/software-layer/pulls?q=is%3Apr+merged%3A%3E2024-01-27+sort%3Aupdated-asc+ The list includes PRs for adding software to EESSI as well as some for updating the code-base (or both). Below we list the PRs in the order in which they were merged (note, it seems some where updated after they got merged) and classify them as code, software and other. Then for each of the PRs to handle (primarily code and other) we describe the actions performed for syncing NESSI (TODO double-check PRs which require changes on CVMFS and mark them with cvmfs):

TO BE DONE

foss/2023a

• software {2023.06}[2023a] Modflow 6.4.4 and deps EESSI/software-layer#522
  • @trz42 looks into this
  • see {2023.06}[foss/2023a] Modflow v6.4.4 and deps #336
  • build in parts
    • see {2023.06}[foss/2023a] SuperLU_DIST v8.1.2 #339
    • see {2023.06}[foss/2023a] Hypre/2.29.0, SuiteSparse/7.1.0 and netCDF-Fortran/4.6.1 #341

foss/2023b

• software {2023.06}[GCCcore/13.2.0] Qt5 v5.15.13 EESSI/software-layer#462
  • @trz42 looks into this
  • see {2023.06}[foss/2023b] Qt5 v5.15.13 #332
  • see partial build via {2023.06}[foss/2023b] HarfBuzz v8.2.2 #337
  • see partial build via {2023.06}[foss/2023b] Mesa v23.1.9 #338
  • see partial build via {2023.06}[foss/2023b] Qt5 dependencies #340

mixed

• software CUDA (POSTPONE ?) {2023.06}[gompi/2023a,gompi/2023b] OSU Microbenchmarks v 7.1.1, OSU Microbenchmarks v 7.2 w/ CUDA 12.1.1 EESSI/software-layer#487

---

DONE

• other bump action versions EESSI/software-layer#465
  • already implemented via {2023.06}[foss/2022b] GLib/2.75.0 #258
• software {2023.06}{GCCcore/12.3.0} BWA 0.7.17.20220923 EESSI/software-layer#468
  • see {2023.06}[foss/2023a] BWA v0.7.17-20220923 #321
• code + software + other {2023.06}[foss/2022b] GDAL 3.6.2 EESSI/software-layer#419
  • code + other addressed in add pre_test hook for dealing with failing Highway tests #297
  • see {2023.06}[foss/2022b] GDAL v3.6.2 #322 for software part
• software {2023.06}[foss/2023a] snakemake 8.4.2 EESSI/software-layer#474
  • already in the software stack
• code (build and deploy required) add Lmod hook to set $OMPI_MCA_btl to ^smcuda when loading OpenMPI module to work around bug EESSI/software-layer#473
  • handled in add Lmod hook to set to ^smcuda when loading OpenMPI module to work around bug + renaming #293 (This is a temporary fix, some changes in the create_lmodrc.py will be reverted in upcoming PRs, but the naming changes will remain.)
• code + software {2023.06}[foss/2023a] LAMMPS 2Aug2023 EESSI/software-layer#472
  • combine withcode fix hook for LAMMPS to filter out ScaFaCoS and tbb dependencies on aarch64/* EESSI/software-layer#477
  • Need to apply the following two commits:
    • EESSI@8151ef1
    • EESSI@4ec7c70
  • First, only apply code changes (eb_hooks.py changes by the above two commits), see add hook for LAMMPS #298
  • see {2023.06}[foss/2023a] LAMMPS v2Aug2023_update2 #323
    • required some tricks for aarch64/generic: setting kokkos_arch to 'ARMV80' and building with deb10 container
• software {2023.06}[foss/2023a] matplotlib v3.7.2 + PyQt5 v5.15.10 EESSI/software-layer#476
  • see {2023.06}[foss/2023a] PyQt5 v5.15.10 #324
• software {2023.06}[foss/2022b] waLBerla v6.1 EESSI/software-layer#478
  • see {2023.06}[foss/2022b] waLBerla v6.1 #327
• software {2023.06}[2023a] JupyterNotebook v7.0.2 EESSI/software-layer#495
  • see {2023.06}[foss/2023a] JupyterNotebook v7.0.2 #326
• software {2023.06}[foss/2022b] ImageMagick v7.1.0-53 EESSI/software-layer#470
  • already in NESSI/2023.06
• software {2023.06}[foss/2023a] Z3 4.12.2 EESSI/software-layer#502
  • already in NESSI/2023.06
• code Improve error reported by bot when using unmerged EasyBuild PR in easystack file EESSI/software-layer#494
  • need to include fix from fix check_missing_installations.sh when unmerged PR is used EESSI/software-layer#506
  • need to include fix for issue logic to determine from-pr fails when .diff files includes extra 'from-pr' EESSI/software-layer#536
  • need to include fix from use apptainer version < 1.3.0 #303 to ensure CI runs through
    • this includes the changes for stick to Apptainer < 1.3.0 for running CI workflows EESSI/software-layer#516
  • see improve error reporting when EasyBuild PR is not merged #299
    • merged despite failing CI (opened PR debug failing CI #304 to investigate issue)
  • accompanying test PRs
    • test case A for PR299 #300
    • test case C for PR299 #301
    • test case B for PR299 #302
• software {2023.06}[foss/2023a] dask v2023.9.2 EESSI/software-layer#481
  • already in NESSI/2023.06
• code Make check_missing_installations.sh check against develop branch of EasyBuild EESSI/software-layer#486
  • already done in Make check_missing_installations.sh against develop branch #285
• code Temporary fix for CI by pinning archspec version EESSI/software-layer#482
  • already done in Temporary fix for CI by pinning archspec version #283
• code + cvmfs (normal ingest) Workaround for LMOD only allowing a single hook to be registered, which made our first hook unused EESSI/software-layer#490
  • see workaround for LMOD to register two hooks #305
• software {2023.06}[gompi/2023b] CDO v2.2.2 EESSI/software-layer#492
  • see {2023.06}[foss/2023b] CDO v2.2.2 #325
• code fix check_missing_installations.sh when unmerged PR is used EESSI/software-layer#506
  • already implemented together with Improve error reported by bot when using unmerged EasyBuild PR in easystack file EESSI/software-layer#494
• software {2023.06}[2023a] OpenJPEG 2.5.0 EESSI/software-layer#511
  • already in NESSI/2023.06
• code stick to Apptainer < 1.3.0 for running CI workflows EESSI/software-layer#516
  • done via use apptainer version < 1.3.0 #303 which was needed to let CI in improve error reporting when EasyBuild PR is not merged #299 succeed
• software {2023.06}[2023b] LittleCMS v2.15, giflib v5.2.1, OpenJPEG v2.5.0, libwebp v1.3.2 EESSI/software-layer#513
  • already in NESSI/2023.06
• code + cvmfs (+ CUDA ?) Always set $LMOD_RC (even when EESSI_BASIC_ENV is set), and make sure loading CUDA apps always work when building EESSI/software-layer#475
  • see Always use LMOD_RC file, and make sure loading CUDA apps always works when building #307
    • needs two changes from 496 to add changes to init/ scripts to tarball
    • also needs to change ingest-tarball.sh script on Stratum 0 (see Disable tarball content type check in ingestion script EESSI/filesystem-layer#186)
• code + cvmfs Split off the Lmod hooks from lmodrc.lua into a seperate SitePackage.lua file EESSI/software-layer#496
  • NOTE, need to handle EESSI PR 475 first.
  • see split Lmod hooks #306
• code + cvmfs configure Lmod via $LMOD_CONFIG_DIR and $LMOD_PACKAGE_PATH + don't set $LMOD_RC EESSI/software-layer#524
  • see configure Lmod via $LMOD_CONFIG_DIR and $LMOD_PACKAGE_PATH + don't set $LMOD_RC #308
• code + filesystem-layer Don't update Lmod cache in software installation script EESSI/software-layer#512
  • see remove Lmod cache update #309
  • requires action on Stratum-0, see Add script for updating Lmod caches EESSI/filesystem-layer#175
    • DONE
  • should make another PR (see {2023.06}[foss/2023a] parallel v20230722 (also: test remove Lmod update cache) #310) to verify
    • if tarballs don't contain Lmod cache updates, and
    • Lmod caches are updated during ingestion
• code + cvmfs-manually Add functionality for rebuilding software: try it on OpenMPI 4.1.x to fix smcuda issue EESSI/software-layer#488
  • see functionality for rebuilding software: try it on OpenMPI 4.1.x to fix smcuda issue #311 and functionality for rebuilding software #312
  • NOTE, only added code. Actually rebuilding doesn't work currently (see issue https://github.com/NorESSI/software-layer/issue/313)
  • {2023.06}[system] EasyBuild v4.9.1 #314 was used as a test to ensure that functionality for rebuilding software #312 doesn't break building software.
  • can probably combined with Fix rebuild check in bot/build.sh script, and always resume from the correct temporary directory EESSI/software-layer#518
• code Fix rebuild check in bot/build.sh script, and always resume from the correct temporary directory EESSI/software-layer#518
  • see functionality for rebuilding software: try it on OpenMPI 4.1.x to fix smcuda issue #311 and functionality for rebuilding software #312
  • can probably combined with Add functionality for rebuilding software: try it on OpenMPI 4.1.x to fix smcuda issue EESSI/software-layer#488
• software + cvmfs-manually Rebuild GCCcore-12.3.0 and 13.2.0 to fix aarch64 vectorization issue EESSI/software-layer#521
  • for rebuilding we use similar mechanism as in EESSI (easystacks/software.eessi.io/2023.06/rebuilds/20240328-eb-4.9.0-GCCcore-fix-aarch64-vectorization.yml), but we need to remove the software first from the CVMFS repo
  • see {2023.06}[foss/2023a,foss/2023b] ignore fakeroot and do not remove packages & rebuild GCCcore 12.3.0 / 13.2.0 #318
• code skip installation of CUDA into host_injections when testing install_software_layer.sh script EESSI/software-layer#528
  • see skip installation of CUDA into host_injections when testing install_software_layer.sh #315
• code + cvmfs Allow defining lmod hooks in host injections EESSI/software-layer#525
  • see Allow defining lmod hooks in host injections #316
• code let bot run EESSI test suite during test phase (part of bot build procedure) EESSI/software-layer#467
  • let bot run EESSI test suite after building #317
• code CAN BE SKIPPED ? (included in 467) add scripts to run test suite after build job has finished EESSI/software-layer#366
  • see let bot run EESSI test suite after building #317
• code + cvmfs (+ CUDA ?) Always set $LMOD_RC (even when EESSI_BASIC_ENV is set), and make sure loading CUDA apps always work when building EESSI/software-layer#475
  • seems obsolete / changes already in place
• code + software {2023.06}[2023a] Highway v1.0.4 EESSI/software-layer#529
  • see {2023.06}[2023a] Highway v1.0.4 #320
  • turned out to be a pure code update because the package (Highway) was already in the stack
• code (combine with PR 529?) Fix Highway hook: change optarch in pre_prepare_hook and reset it in post_prepare_hook EESSI/software-layer#532
  • combine with 529 but should rebuild GCC and OpenMPI first
  • see {2023.06}[2023a] Highway v1.0.4 #320
  • turned out to be a pure code update because the package (Highway) was already in the stack
• software {2023.06}[foss/2022b] R v4.2.2 EESSI/software-layer#452
  • see {2023.06}[foss/2022b] R v4.2.2 #328
• software {2023.06}[foss/2023a] R-bundle-CRAN 2023.12 EESSI/software-layer#428
  • see {2023.06}[foss/2023a] R-bundle-CRAN 2023.12 #290
• software {2023.06}[2023a] ImageMagick v7.1.1-15 EESSI/software-layer#498
  • possibly already included in {2023.06}[foss/2023a] R-bundle-CRAN 2023.12 #290
• software {2023.06}[foss/2023a] ELPA 2023.05.001 + libxc 6.2.2 EESSI/software-layer#508
  • already in NESSI/2023.06
• software {2023.06}[foss/2022b] WRF v4.4.1 EESSI/software-layer#480
  • see {2023.06}[foss/2022b] WRF v4.4.1 #330
• software {2023.06}[foss/2023a] OpenFOAM v10 EESSI/software-layer#404
  • see {2023.06}[foss/2023a] OpenFOAM v10 #331
• software {2023.06}[2023a] PyOpenGL 3.1.7 EESSI/software-layer#503
  • see {2023.06}[foss/2023a] PyOpenGL v3.1.7 #333
• software {2023.06}[2023a] ipympl 0.9.3 EESSI/software-layer#505
  • see {2023.06}[foss/2023a] ipympl v0.9.3 #334
• software {2023.06}[GCCcore/13.2.0] Wayland v1.22.0 EESSI/software-layer#520
  • see {2023.06}[foss/2023b] Wayland v1.22.0 #335
• code + software {2023.06}[foss/2023b] WSClean 3.4 + DP3 6.0 EESSI/software-layer#479
  • adds a hook for casacore
  • will add the hook later together with the software
  • see {2023.06}[foss/2023b] DP3-6.0, WSClean 3.4 and deps #329

---

Notes

• April 11: most code changes done
• April 12/13:
  1. rebuild GCC ✅
  2. rebuild OpenMPI ✅
  • for rebuilding we use similar mechanism as in EESSI (rebuilds/eessi...), but we need to remove the software first from the CVMFS repo
• April 14:
  1. do 529+532 ✅
• afterwards missing softwares
  • most done by April 18
  • the last packages fail for different reasons on different architectures and by different bot instances
    • if possible split PRs or build some dependencies to make some progress towards completion of the builds
• also check new PRs: https://github.com/EESSI/software-layer/pulls?q=is%3Apr+merged%3A%3E2024-04-07+sort%3Aupdated-asc
• do a final file-by-file check for differences and document them