functionality for rebuilding software #312

trz42 · 2024-04-10T18:56:58Z

This PR is a stripped-down version of #311. It only contains the code we can retain (removed changes to create_lmodsitepackage.py which would create a new SitePackage.lua). It also doesn't rebuild OpenMPI because rebuilding currently does not work as envisioned in #311.

This PR syncs NESSI with EESSI. Particularly it combines most of EESSI#488 and EESSI#518.

We test only if the build procedure still works after adding rebuild capabilities. Nothing should be put into a tarball that requires ingesting.

nessi-bot · 2024-04-10T18:57:02Z

Instance AWS-MC-NESSI is configured to build:

arch x86_64/generic for repo nessi-2022.11-swl-deb10
arch x86_64/generic for repo nessi-2023.06-cl
arch x86_64/generic for repo nessi-2023.06-swl-deb10
arch x86_64/generic for repo nessi-2023.06-swl-deb11
arch x86_64/intel/skylake_avx512 for repo nessi-2022.11-swl-deb10
arch x86_64/intel/skylake_avx512 for repo nessi-2023.06-cl
arch x86_64/intel/skylake_avx512 for repo nessi-2023.06-swl-deb10
arch x86_64/intel/skylake_avx512 for repo nessi-2023.06-swl-deb11
arch x86_64/amd/zen2 for repo nessi-2022.11-swl-deb10
arch x86_64/amd/zen2 for repo nessi-2023.06-cl
arch x86_64/amd/zen2 for repo nessi-2023.06-swl-deb10
arch x86_64/amd/zen2 for repo nessi-2023.06-swl-deb11
arch aarch64/generic for repo nessi-2022.11-swl-deb10
arch aarch64/generic for repo nessi-2023.06-cl
arch aarch64/generic for repo nessi-2023.06-swl-deb10
arch aarch64/generic for repo nessi-2023.06-swl-deb11

nessi-bot · 2024-04-10T18:57:03Z

Instance eX3-NESSI is configured to build:

arch x86_64/amd/zen2 for repo nessi-2023.06-cl
arch x86_64/amd/zen2 for repo nessi-2023.06-swl-deb10
arch x86_64/amd/zen2 for repo nessi-2023.06-swl-deb11
arch aarch64/generic for repo nessi-2023.06-cl
arch aarch64/generic for repo nessi-2023.06-swl-deb10
arch aarch64/generic for repo nessi-2023.06-swl-deb11

nessi-bot · 2024-04-10T18:57:03Z

Instance Fram-NESSI is configured to build:

arch x86_64/generic for repo nessi-2023.06-cl
arch x86_64/generic for repo nessi-2023.06-swl-deb10
arch x86_64/generic for repo nessi-2023.06-swl-deb11
arch x86_64/intel/broadwell for repo nessi-2023.06-cl
arch x86_64/intel/broadwell for repo nessi-2023.06-swl-deb10
arch x86_64/intel/broadwell for repo nessi-2023.06-swl-deb11

nessi-bot · 2024-04-10T18:57:03Z

Instance Saga-NESSI is configured to build:

arch x86_64/intel/skylake_avx512 for repo nessi-2023.06-cl
arch x86_64/intel/skylake_avx512 for repo nessi-2023.06-swl-deb10
arch x86_64/intel/skylake_avx512 for repo nessi-2023.06-swl-deb11
arch x86_64/intel/broadwell for repo nessi-2023.06-cl
arch x86_64/intel/broadwell for repo nessi-2023.06-swl-deb10
arch x86_64/intel/broadwell for repo nessi-2023.06-swl-deb11
arch x86_64/generic for repo nessi-2023.06-cl
arch x86_64/generic for repo nessi-2023.06-swl-deb10
arch x86_64/generic for repo nessi-2023.06-swl-deb11

trz42 · 2024-04-10T18:57:18Z

bot: build inst:Fram-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/broadwell
bot: build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/skylake
bot: build inst:eX3-NESSI repo:nessi-2023.06-swl-deb11 arch:aarch64/generic
bot: build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2
bot: build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/generic

nessi-bot · 2024-04-10T18:57:21Z

Updates by the bot instance eX3-NESSI (click for details)

received bot command build inst:Fram-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/broadwell from trz42
- expanded format: build instance:Fram-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell
received bot command build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/skylake from trz42
- expanded format: build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake
received bot command build inst:eX3-NESSI repo:nessi-2023.06-swl-deb11 arch:aarch64/generic from trz42
- expanded format: build instance:eX3-NESSI repository:nessi-2023.06-swl-deb11 architecture:aarch64/generic
received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/generic from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic
handling command build instance:Fram-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell resulted in:
- no jobs were submitted
handling command build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake resulted in:
- no jobs were submitted
handling command build instance:eX3-NESSI repository:nessi-2023.06-swl-deb11 architecture:aarch64/generic resulted in:
- submitted job 166834, for details & status see functionality for rebuilding software #312 (comment)
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic resulted in:
- no jobs were submitted

nessi-bot · 2024-04-10T18:57:21Z

Updates by the bot instance AWS-MC-NESSI (click for details)

received bot command build inst:Fram-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/broadwell from trz42
- expanded format: build instance:Fram-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell
received bot command build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/skylake from trz42
- expanded format: build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake
received bot command build inst:eX3-NESSI repo:nessi-2023.06-swl-deb11 arch:aarch64/generic from trz42
- expanded format: build instance:eX3-NESSI repository:nessi-2023.06-swl-deb11 architecture:aarch64/generic
received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/generic from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic
handling command build instance:Fram-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell resulted in:
- no jobs were submitted
handling command build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake resulted in:
- no jobs were submitted
handling command build instance:eX3-NESSI repository:nessi-2023.06-swl-deb11 architecture:aarch64/generic resulted in:
- no jobs were submitted
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- submitted job 9278, for details & status see functionality for rebuilding software #312 (comment)
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic resulted in:
- submitted job 9279, for details & status see functionality for rebuilding software #312 (comment)

nessi-bot · 2024-04-10T18:57:21Z

Updates by the bot instance Saga-NESSI (click for details)

received bot command build inst:Fram-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/broadwell from trz42
- expanded format: build instance:Fram-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell
received bot command build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/skylake from trz42
- expanded format: build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake
received bot command build inst:eX3-NESSI repo:nessi-2023.06-swl-deb11 arch:aarch64/generic from trz42
- expanded format: build instance:eX3-NESSI repository:nessi-2023.06-swl-deb11 architecture:aarch64/generic
received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/generic from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic
handling command build instance:Fram-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell resulted in:
- no jobs were submitted
handling command build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake resulted in:
- submitted job 11139840, for details & status see functionality for rebuilding software #312 (comment)
handling command build instance:eX3-NESSI repository:nessi-2023.06-swl-deb11 architecture:aarch64/generic resulted in:
- no jobs were submitted
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic resulted in:
- no jobs were submitted

nessi-bot · 2024-04-10T18:57:22Z

Updates by the bot instance Fram-NESSI (click for details)

received bot command build inst:Fram-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/broadwell from trz42
- expanded format: build instance:Fram-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell
received bot command build inst:Saga-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/intel/skylake from trz42
- expanded format: build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake
received bot command build inst:eX3-NESSI repo:nessi-2023.06-swl-deb11 arch:aarch64/generic from trz42
- expanded format: build instance:eX3-NESSI repository:nessi-2023.06-swl-deb11 architecture:aarch64/generic
received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:zen2 from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2
received bot command build inst:AWS-MC-NESSI repo:nessi-2023.06-swl-deb11 arch:x86_64/generic from trz42
- expanded format: build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic
handling command build instance:Fram-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/broadwell resulted in:
- submitted job 5764518, for details & status see functionality for rebuilding software #312 (comment)
handling command build instance:Saga-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/intel/skylake resulted in:
- no jobs were submitted
handling command build instance:eX3-NESSI repository:nessi-2023.06-swl-deb11 architecture:aarch64/generic resulted in:
- no jobs were submitted
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:zen2 resulted in:
- no jobs were submitted
handling command build instance:AWS-MC-NESSI repository:nessi-2023.06-swl-deb11 architecture:x86_64/generic resulted in:
- no jobs were submitted

nessi-bot · 2024-04-10T18:57:27Z

New job on instance Fram-NESSI for architecture x86_64-intel-broadwell for repository nessi-2023.06-swl-deb11 in job dir /cluster/projects/nn9992k/pilot.nessi.no/jobs/2024.04/pr_312/5764518

date	job status	comment
Apr 10 18:57:26 UTC 2024	submitted	job id `5764518` awaits release by job manager
Apr 10 18:57:39 UTC 2024	released	job awaits launch by Slurm scheduler
Apr 10 18:58:40 UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-5764518.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts No artefacts were created or found.
Apr 10 18:58:40 UTC 2024	test result	(no tests yet)

nessi-bot · 2024-04-10T18:57:27Z

New job on instance AWS-MC-NESSI for architecture x86_64-amd-zen2 for repository nessi-2023.06-swl-deb11 in job dir /project/def-nessi/SHARED/jobs/2024.04/pr_312/9278

date	job status	comment
Apr 10 18:57:26 UTC 2024	submitted	job id `9278` awaits release by job manager
Apr 10 18:57:56 UTC 2024	released	job awaits launch by Slurm scheduler
Apr 10 19:04:00 UTC 2024	running	job `9278` is running
Apr 10 19:05:03 UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-9278.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts No artefacts were created or found.
Apr 10 19:05:03 UTC 2024	test result	(no tests yet)

nessi-bot · 2024-04-10T18:57:31Z

New job on instance eX3-NESSI for architecture aarch64-generic for repository nessi-2023.06-swl-deb11 in job dir /home/thomarob/pilot.nessi.no/jobs/2024.04/pr_312/166834

date	job status	comment
Apr 10 06:57:30 PM UTC 2024	submitted	job id `166834` awaits release by job manager
Apr 10 06:57:44 PM UTC 2024	released	job awaits launch by Slurm scheduler
Apr 10 06:58:45 PM UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-166834.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts No artefacts were created or found.
Apr 10 06:58:45 PM UTC 2024	test result	(no tests yet)

nessi-bot · 2024-04-10T18:57:31Z

New job on instance AWS-MC-NESSI for architecture x86_64-generic for repository nessi-2023.06-swl-deb11 in job dir /project/def-nessi/SHARED/jobs/2024.04/pr_312/9279

date	job status	comment
Apr 10 18:57:30 UTC 2024	submitted	job id `9279` awaits release by job manager
Apr 10 18:57:58 UTC 2024	released	job awaits launch by Slurm scheduler
Apr 10 19:04:02 UTC 2024	running	job `9279` is running
Apr 10 19:05:05 UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-9279.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts No artefacts were created or found.
Apr 10 19:05:05 UTC 2024	test result	(no tests yet)

nessi-bot · 2024-04-10T18:57:46Z

New job on instance Saga-NESSI for architecture x86_64-intel-skylake_avx512 for repository nessi-2023.06-swl-deb11 in job dir /cluster/projects/nn9992k/pilot.nessi.no/jobs/2024.04/pr_312/11139840

date	job status	comment
Apr 10 06:57:46 PM UTC 2024	submitted	job id `11139840` awaits release by job manager
Apr 10 06:58:10 PM UTC 2024	released	job awaits launch by Slurm scheduler
Apr 10 06:59:11 PM UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-11139840.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts No artefacts were created or found.
Apr 10 06:59:11 PM UTC 2024	test result	(no tests yet)

trz42 · 2024-04-10T19:20:31Z

Despite the seemingly failed builds everything is fine. The FAILURE was only reported because there was no software built at all and hence the architecture specific directory from which a tarball would be created did not exist.

For example, for Broadwell we have the following in the slurm output:

Software directory overlay /tmp/overlay-upper/versions/2023.06/software/linux/x86_64/intel/broadwell does not exist?!

poksumdo

Not fully satisfactory, but we need to move on. Should keep #311 updated with nessi.no-2023.06 branch.

truib added 9 commits April 9, 2024 20:47

removed a few extra whitespaces in empty lines

fedf9e3

new script to remove a software

0125608

bot/build.sh from EESSI PR 488

42d79af

create_lmodsitepackage.py from EESSI PR 488

7582f5a

eessi_container.sh from EESSI PR 488

4951647

easystack file for rebuilding three installations of OpenMPI

1e229d7

add modifications from EESSI PR 518

0083ddf

modifications to bot/build.sh from EESSI PR 518

172242b

revert changes to SitePackage.lua + don't rebuild OpenMPI

4afaaad

trz42 added rebuild NESSI/2023.06 sync-with-EESSI labels Apr 10, 2024

trz42 mentioned this pull request Apr 10, 2024

fix/add rebuild capability to NESSI #313

Open

poksumdo approved these changes Apr 10, 2024

View reviewed changes

poksumdo merged commit 32031cb into NorESSI:nessi.no-2023.06 Apr 10, 2024

This was referenced Apr 10, 2024

{2023.06}[system] EasyBuild v4.9.1 #314

Merged

Syncing code-base and CVMFS repository with EESSI (April 2024) #294

Open

{2023.06}[foss/2023a,foss/2023b] ignore fakeroot and do not remove packages & rebuild GCCcore 12.3.0 / 13.2.0 #318

Merged

functionality for rebuilding software #312

functionality for rebuilding software #312

Uh oh!

Conversation

trz42 commented Apr 10, 2024

Uh oh!

nessi-bot bot commented Apr 10, 2024

Uh oh!

nessi-bot bot commented Apr 10, 2024

Uh oh!

nessi-bot bot commented Apr 10, 2024

Uh oh!

nessi-bot bot commented Apr 10, 2024

Uh oh!

trz42 commented Apr 10, 2024

Uh oh!

nessi-bot bot commented Apr 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nessi-bot bot commented Apr 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nessi-bot bot commented Apr 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nessi-bot bot commented Apr 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nessi-bot bot commented Apr 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nessi-bot bot commented Apr 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nessi-bot bot commented Apr 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nessi-bot bot commented Apr 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nessi-bot bot commented Apr 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trz42 commented Apr 10, 2024

Uh oh!

poksumdo left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nessi-bot bot commented Apr 10, 2024 •

edited

Loading

nessi-bot bot commented Apr 10, 2024 •

edited

Loading

nessi-bot bot commented Apr 10, 2024 •

edited

Loading

nessi-bot bot commented Apr 10, 2024 •

edited

Loading

nessi-bot bot commented Apr 10, 2024 •

edited

Loading

nessi-bot bot commented Apr 10, 2024 •

edited

Loading

nessi-bot bot commented Apr 10, 2024 •

edited

Loading

nessi-bot bot commented Apr 10, 2024 •

edited

Loading

nessi-bot bot commented Apr 10, 2024 •

edited

Loading