Allow defining lmod hooks in host injections #525

casparvl · 2024-03-29T15:50:27Z

Fixes #456

For proper testing, we want this fix to be deployed first or you need to make sure to unset LMOD_RC (after sourcing the EESSI environment). Otherwise, you'll find that no matter what local change you make, you are always executing the EESSI hook (and only the EESSI hook), since the LMOD_RC is still set :)

Because our current lmod version doesn't support registering multiple hooks yet (only supported from version 8.7.36 onwards, see here), the person implementing the site-specific hook needs to define a combined function that also calls the EESSI hook (if he/she wants to append to the EESSI hook's functionality, of course you can always simply overwrite it).

The easiest way to test this PR is to do the following:

python3 create_lmodsitepackage.py .
export LMOD_PACKAGE_PATH=$PWD/.lmod

This generates the SitePackage.py in your current dir, in a .lmod subdir, then point the LMOD_PACKAGE_PATH to it. You can check that it gets picked up correctly by running module --config and looking for the Site Pkg location field. If you want, you can add some LmodMessage("somemessage") messages to this SitePackage.py to see when certain parts of it are executed.

Then, put two dummy hooks in place in the host_injections prefix, e.g.:

$ cat /cvmfs/software.eessi.io/host_injections/2023.06/.lmod/SitePackage.lua
require("strict")
local hook = require("Hook")

LmodMessage("Load host SitePackage.lua")

function site_specific_load_hook(t)
    local frameStk  = require("FrameStk"):singleton()
    local mt        = frameStk:mt()
    local simpleName = string.match(t.modFullName, "(.-)/")
    LmodMessage("Site-specific load hook called, loading " .. simpleName)
end

local function combined_load_hook(t)
    LmodMessage("Defining combined load hook")
    -- Assuming this SitePackage.lua was called from EESSI's SitePackage.lua
    -- the eessi_load_hook function is defined, and should be called
    if eessi_load_hook ~= nil then
        eessi_load_hook(t)
    end
    site_specific_load_hook(t)
end

hook.register("load", combined_load_hook)
$ cat /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen2/.lmod/SitePackage.lua
require("strict")
local hook = require("Hook")

LmodMessage("Load architecture-specific SitePackage.lua")

local function architecture_specific_load_hook(t)
    local frameStk  = require("FrameStk"):singleton()
    local mt        = frameStk:mt()
    local simpleName = string.match(t.modFullName, "(.-)/")
    LmodMessage("Architecture-specific load hook called, loading " .. simpleName)
end

local function combined_load_hook(t)
    LmodMessage("Defining combined load hook in arch-specific SitePackage.lua")
    -- Assuming this SitePackage.lua was called from EESSI's SitePackage.lua
    -- the eessi_load_hook function is defined, and should be called
    if eessi_load_hook ~= nil then
        eessi_load_hook(t)
    end
    if site_specific_load_hook ~= nil then
        site_specific_load_hook(t)
    end
    architecture_specific_load_hook(t)
end

hook.register("load", combined_load_hook)

Now, to test it:

$ module load GCCcore/12.3.0
Loading EESSI's SitePackage.lua
Load host SitePackage.lua
Load architecture-specific SitePackage.lua
Defining combined load hook in arch-specific SitePackage.lua
EESSI load hook called, loading GCCcore
Site-specific load hook called, loading GCCcore
Architecture-specific load hook called, loading GCCcore

(as you can see, I added some print statements to check the order in which they are executed, and to see if all are executed). IMPORTANT: all the functions that have to be used in other translation units (e.g. the eessi_load_hook and site_specific_load_hook) need to be defined without a local keyword (for obvious reasons :)).

Similarly, if a site wants their host or architecture specific hook to just overwrite the EESSI hook, they could do:

$ cat /cvmfs/software.eessi.io/host_injections/2023.06/.lmod/SitePackage.lua
require("strict")
local hook = require("Hook")

LmodMessage("Load host SitePackage.lua")

local function site_specific_load_hook(t)
    local frameStk  = require("FrameStk"):singleton()
    local mt        = frameStk:mt()
    local simpleName = string.match(t.modFullName, "(.-)/")
    LmodMessage("Site-specific load hook called, loading " .. simpleName)
end

hook.register("load", site_specific_load_hook)

Note that once we have Lmod version 8.7.36 or later, things become much simpler if you want to append:

$ cat /cvmfs/software.eessi.io/host_injections/2023.06/.lmod/SitePackage.lua
require("strict")
local hook = require("Hook")

LmodMessage("Load host SitePackage.lua")

local function site_specific_load_hook(t)
    local frameStk  = require("FrameStk"):singleton()
    local mt        = frameStk:mt()
    local simpleName = string.match(t.modFullName, "(.-)/")
    LmodMessage("Site-specific load hook called, loading " .. simpleName)
end

hook.register("load", site_specific_load_hook, "append")

The big advantage here is that the person implementing the site specific hook does not need to know the name of the EESSI hook in order to append to it.

…hey exist

eessi-bot · 2024-03-29T15:50:31Z

Instance eessi-bot-mc-aws is configured to build:

arch x86_64/generic for repo eessi-hpc.org-2023.06-compat
arch x86_64/generic for repo eessi-hpc.org-2023.06-software
arch x86_64/generic for repo eessi.io-2023.06-compat
arch x86_64/generic for repo eessi.io-2023.06-software
arch x86_64/intel/haswell for repo eessi-hpc.org-2023.06-compat
arch x86_64/intel/haswell for repo eessi-hpc.org-2023.06-software
arch x86_64/intel/haswell for repo eessi.io-2023.06-compat
arch x86_64/intel/haswell for repo eessi.io-2023.06-software
arch x86_64/intel/skylake_avx512 for repo eessi-hpc.org-2023.06-compat
arch x86_64/intel/skylake_avx512 for repo eessi-hpc.org-2023.06-software
arch x86_64/intel/skylake_avx512 for repo eessi.io-2023.06-compat
arch x86_64/intel/skylake_avx512 for repo eessi.io-2023.06-software
arch x86_64/amd/zen2 for repo eessi-hpc.org-2023.06-compat
arch x86_64/amd/zen2 for repo eessi-hpc.org-2023.06-software
arch x86_64/amd/zen2 for repo eessi.io-2023.06-compat
arch x86_64/amd/zen2 for repo eessi.io-2023.06-software
arch x86_64/amd/zen3 for repo eessi-hpc.org-2023.06-compat
arch x86_64/amd/zen3 for repo eessi-hpc.org-2023.06-software
arch x86_64/amd/zen3 for repo eessi.io-2023.06-compat
arch x86_64/amd/zen3 for repo eessi.io-2023.06-software
arch aarch64/generic for repo eessi-hpc.org-2023.06-compat
arch aarch64/generic for repo eessi-hpc.org-2023.06-software
arch aarch64/generic for repo eessi.io-2023.06-compat
arch aarch64/generic for repo eessi.io-2023.06-software
arch aarch64/neoverse_n1 for repo eessi-hpc.org-2023.06-compat
arch aarch64/neoverse_n1 for repo eessi-hpc.org-2023.06-software
arch aarch64/neoverse_n1 for repo eessi.io-2023.06-compat
arch aarch64/neoverse_n1 for repo eessi.io-2023.06-software
arch aarch64/neoverse_v1 for repo eessi-hpc.org-2023.06-compat
arch aarch64/neoverse_v1 for repo eessi-hpc.org-2023.06-software
arch aarch64/neoverse_v1 for repo eessi.io-2023.06-compat
arch aarch64/neoverse_v1 for repo eessi.io-2023.06-software

casparvl · 2024-03-29T16:03:52Z

bot: build repo:eessi.io-2023.06-software arch:aarch64/generic
bot: build repo:eessi.io-2023.06-software arch:aarch64/neoverse_n1
bot: build repo:eessi.io-2023.06-software arch:aarch64/neoverse_v1
bot: build repo:eessi.io-2023.06-software arch:x86_64/generic
bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2
bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3
bot: build repo:eessi.io-2023.06-software arch:x86_64/intel/haswell
bot: build repo:eessi.io-2023.06-software arch:x86_64/intel/skylake_avx512

eessi-bot · 2024-03-29T16:03:55Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:aarch64/generic from casparvl
- expanded format: build repository:eessi.io-2023.06-software architecture:aarch64/generic
received bot command build repo:eessi.io-2023.06-software arch:aarch64/neoverse_n1 from casparvl
- expanded format: build repository:eessi.io-2023.06-software architecture:aarch64/neoverse_n1
received bot command build repo:eessi.io-2023.06-software arch:aarch64/neoverse_v1 from casparvl
- expanded format: build repository:eessi.io-2023.06-software architecture:aarch64/neoverse_v1
received bot command build repo:eessi.io-2023.06-software arch:x86_64/generic from casparvl
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/generic
received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 from casparvl
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2
received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 from casparvl
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3
received bot command build repo:eessi.io-2023.06-software arch:x86_64/intel/haswell from casparvl
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/intel/haswell
received bot command build repo:eessi.io-2023.06-software arch:x86_64/intel/skylake_avx512 from casparvl
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/intel/skylake_avx512
handling command build repository:eessi.io-2023.06-software architecture:aarch64/generic resulted in:
- submitted job 8768, for details & status see Allow defining lmod hooks in host injections #525 (comment)
handling command build repository:eessi.io-2023.06-software architecture:aarch64/neoverse_n1 resulted in:
- submitted job 8769, for details & status see Allow defining lmod hooks in host injections #525 (comment)
handling command build repository:eessi.io-2023.06-software architecture:aarch64/neoverse_v1 resulted in:
- submitted job 8770, for details & status see Allow defining lmod hooks in host injections #525 (comment)
handling command build repository:eessi.io-2023.06-software architecture:x86_64/generic resulted in:
- submitted job 8771, for details & status see Allow defining lmod hooks in host injections #525 (comment)
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 resulted in:
- submitted job 8772, for details & status see Allow defining lmod hooks in host injections #525 (comment)
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 resulted in:
- submitted job 8773, for details & status see Allow defining lmod hooks in host injections #525 (comment)
handling command build repository:eessi.io-2023.06-software architecture:x86_64/intel/haswell resulted in:
- submitted job 8774, for details & status see Allow defining lmod hooks in host injections #525 (comment)
handling command build repository:eessi.io-2023.06-software architecture:x86_64/intel/skylake_avx512 resulted in:
- submitted job 8775, for details & status see Allow defining lmod hooks in host injections #525 (comment)

eessi-bot · 2024-03-29T16:03:58Z

New job on instance eessi-bot-mc-aws for architecture aarch64-generic for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.03/pr_525/8768

date	job status	comment
Mar 29 16:03:57 UTC 2024	submitted	job id `8768` awaits release by job manager
Mar 29 16:04:13 UTC 2024	released	job awaits launch by Slurm scheduler
Mar 29 16:05:26 UTC 2024	running	job `8768` is running
Mar 29 16:14:24 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8768.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-generic-1711728268.tar.gz` size: 0 MiB (2416 bytes) entries: 1 modules under 2023.06/software/linux/aarch64/generic/modules/all no module files in tarball software under 2023.06/software/linux/aarch64/generic/software no software packages in tarball other under 2023.06/software/linux/aarch64/generic `.lmod/SitePackage.lua`
Mar 29 16:14:24 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8768.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot · 2024-03-29T16:04:02Z

New job on instance eessi-bot-mc-aws for architecture aarch64-neoverse_n1 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.03/pr_525/8769

date	job status	comment
Mar 29 16:04:01 UTC 2024	submitted	job id `8769` awaits release by job manager
Mar 29 16:04:14 UTC 2024	released	job awaits launch by Slurm scheduler
Mar 29 16:05:28 UTC 2024	running	job `8769` is running
Mar 29 16:14:26 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8769.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-neoverse_n1-1711728277.tar.gz` size: 0 MiB (2420 bytes) entries: 1 modules under 2023.06/software/linux/aarch64/neoverse_n1/modules/all no module files in tarball software under 2023.06/software/linux/aarch64/neoverse_n1/software no software packages in tarball other under 2023.06/software/linux/aarch64/neoverse_n1 `.lmod/SitePackage.lua`
Mar 29 16:14:26 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8769.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot · 2024-03-29T16:04:05Z

New job on instance eessi-bot-mc-aws for architecture aarch64-neoverse_v1 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.03/pr_525/8770

date	job status	comment
Mar 29 16:04:04 UTC 2024	submitted	job id `8770` awaits release by job manager
Mar 29 16:04:16 UTC 2024	released	job awaits launch by Slurm scheduler
Mar 29 16:05:30 UTC 2024	running	job `8770` is running
Mar 29 16:12:14 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8770.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-neoverse_v1-1711728282.tar.gz` size: 0 MiB (2421 bytes) entries: 1 modules under 2023.06/software/linux/aarch64/neoverse_v1/modules/all no module files in tarball software under 2023.06/software/linux/aarch64/neoverse_v1/software no software packages in tarball other under 2023.06/software/linux/aarch64/neoverse_v1 `.lmod/SitePackage.lua`
Mar 29 16:12:14 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8770.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot · 2024-03-29T16:04:09Z

New job on instance eessi-bot-mc-aws for architecture x86_64-generic for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.03/pr_525/8771

date	job status	comment
Mar 29 16:04:08 UTC 2024	submitted	job id `8771` awaits release by job manager
Mar 29 16:04:20 UTC 2024	released	job awaits launch by Slurm scheduler
Mar 29 16:05:33 UTC 2024	running	job `8771` is running
Mar 29 16:20:47 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8771.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-generic-1711728280.tar.gz` size: 0 MiB (2412 bytes) entries: 1 modules under 2023.06/software/linux/x86_64/generic/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/generic/software no software packages in tarball other under 2023.06/software/linux/x86_64/generic `.lmod/SitePackage.lua`
Mar 29 16:20:47 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8771.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot · 2024-03-29T16:04:12Z

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.03/pr_525/8772

date	job status	comment
Mar 29 16:04:12 UTC 2024	submitted	job id `8772` awaits release by job manager
Mar 29 16:04:18 UTC 2024	released	job awaits launch by Slurm scheduler
Mar 29 16:05:31 UTC 2024	running	job `8772` is running
Mar 29 16:22:51 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8772.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1711728270.tar.gz` size: 0 MiB (2405 bytes) entries: 1 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `.lmod/SitePackage.lua`
Mar 29 16:22:51 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8772.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot · 2024-03-29T16:04:16Z

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen3 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.03/pr_525/8773

date	job status	comment
Mar 29 16:04:15 UTC 2024	submitted	job id `8773` awaits release by job manager
Mar 29 16:05:22 UTC 2024	released	job awaits launch by Slurm scheduler
Mar 29 16:06:38 UTC 2024	running	job `8773` is running
Mar 29 16:16:34 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8773.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen3-1711728333.tar.gz` size: 0 MiB (2409 bytes) entries: 1 modules under 2023.06/software/linux/x86_64/amd/zen3/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen3/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen3 `.lmod/SitePackage.lua`
Mar 29 16:16:34 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8773.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot · 2024-03-29T16:04:20Z

New job on instance eessi-bot-mc-aws for architecture x86_64-intel-haswell for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.03/pr_525/8774

date	job status	comment
Mar 29 16:04:19 UTC 2024	submitted	job id `8774` awaits release by job manager
Mar 29 16:05:23 UTC 2024	released	job awaits launch by Slurm scheduler
Mar 29 16:06:40 UTC 2024	running	job `8774` is running
Mar 29 16:20:49 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8774.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-intel-haswell-1711728337.tar.gz` size: 0 MiB (2417 bytes) entries: 1 modules under 2023.06/software/linux/x86_64/intel/haswell/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/intel/haswell/software no software packages in tarball other under 2023.06/software/linux/x86_64/intel/haswell `.lmod/SitePackage.lua`
Mar 29 16:20:49 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8774.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

eessi-bot · 2024-03-29T16:04:23Z

New job on instance eessi-bot-mc-aws for architecture x86_64-intel-skylake_avx512 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.03/pr_525/8775

date	job status	comment
Mar 29 16:04:22 UTC 2024	submitted	job id `8775` awaits release by job manager
Mar 29 16:05:25 UTC 2024	released	job awaits launch by Slurm scheduler
Mar 29 16:06:42 UTC 2024	running	job `8775` is running
Mar 29 16:19:45 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8775.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-1711728334.tar.gz` size: 0 MiB (2422 bytes) entries: 1 modules under 2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/intel/skylake_avx512/software no software packages in tarball other under 2023.06/software/linux/x86_64/intel/skylake_avx512 `.lmod/SitePackage.lua`
Mar 29 16:19:45 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8775.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

create_lmodsitepackage.py

boegel · 2024-04-02T15:49:14Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/generic
bot: build repo:eessi.io-2023.06-software arch:x86_64/intel/haswell
bot: build repo:eessi.io-2023.06-software arch:x86_64/intel/skylake_avx512
bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2
bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3
bot: build repo:eessi.io-2023.06-software arch:aarch64/generic
bot: build repo:eessi.io-2023.06-software arch:aarch64/neoverse_n1
bot: build repo:eessi.io-2023.06-software arch:aarch64/neoverse_v1

eessi-bot · 2024-04-02T15:49:16Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/generic from boegel
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/generic
received bot command build repo:eessi.io-2023.06-software arch:x86_64/intel/haswell from boegel
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/intel/haswell
received bot command build repo:eessi.io-2023.06-software arch:x86_64/intel/skylake_avx512 from boegel
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/intel/skylake_avx512
received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 from boegel
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2
received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen3 from boegel
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3
received bot command build repo:eessi.io-2023.06-software arch:aarch64/generic from boegel
- expanded format: build repository:eessi.io-2023.06-software architecture:aarch64/generic
received bot command build repo:eessi.io-2023.06-software arch:aarch64/neoverse_n1 from boegel
- expanded format: build repository:eessi.io-2023.06-software architecture:aarch64/neoverse_n1
received bot command build repo:eessi.io-2023.06-software arch:aarch64/neoverse_v1 from boegel
- expanded format: build repository:eessi.io-2023.06-software architecture:aarch64/neoverse_v1
handling command build repository:eessi.io-2023.06-software architecture:x86_64/generic resulted in:
- submitted job 8918, for details & status see Allow defining lmod hooks in host injections #525 (comment)
handling command build repository:eessi.io-2023.06-software architecture:x86_64/intel/haswell resulted in:
- submitted job 8919, for details & status see Allow defining lmod hooks in host injections #525 (comment)
handling command build repository:eessi.io-2023.06-software architecture:x86_64/intel/skylake_avx512 resulted in:
- submitted job 8920, for details & status see Allow defining lmod hooks in host injections #525 (comment)
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 resulted in:
- submitted job 8921, for details & status see Allow defining lmod hooks in host injections #525 (comment)
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen3 resulted in:
- submitted job 8922, for details & status see Allow defining lmod hooks in host injections #525 (comment)
handling command build repository:eessi.io-2023.06-software architecture:aarch64/generic resulted in:
- submitted job 8923, for details & status see Allow defining lmod hooks in host injections #525 (comment)
handling command build repository:eessi.io-2023.06-software architecture:aarch64/neoverse_n1 resulted in:
- submitted job 8924, for details & status see Allow defining lmod hooks in host injections #525 (comment)
handling command build repository:eessi.io-2023.06-software architecture:aarch64/neoverse_v1 resulted in:
- submitted job 8925, for details & status see Allow defining lmod hooks in host injections #525 (comment)

eessi-bot · 2024-04-02T15:49:20Z

New job on instance eessi-bot-mc-aws for architecture x86_64-generic for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.04/pr_525/8918

date	job status	comment
Apr 02 15:49:19 UTC 2024	submitted	job id `8918` awaits release by job manager
Apr 02 15:49:50 UTC 2024	released	job awaits launch by Slurm scheduler
Apr 02 15:55:15 UTC 2024	running	job `8918` is running
Apr 02 16:10:58 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8918.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-generic-1712073286.tar.gz` size: 0 MiB (2405 bytes) entries: 1 modules under 2023.06/software/linux/x86_64/generic/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/generic/software no software packages in tarball other under 2023.06/software/linux/x86_64/generic `.lmod/SitePackage.lua`
Apr 02 16:10:58 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8918.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
Apr 02 17:24:58 UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-generic-1712073286.tar.gz` to S3 bucket succeeded

eessi-bot · 2024-04-02T15:49:24Z

New job on instance eessi-bot-mc-aws for architecture x86_64-intel-haswell for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.04/pr_525/8919

date	job status	comment
Apr 02 15:49:23 UTC 2024	submitted	job id `8919` awaits release by job manager
Apr 02 15:49:52 UTC 2024	released	job awaits launch by Slurm scheduler
Apr 02 15:55:17 UTC 2024	running	job `8919` is running
Apr 02 16:10:59 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8919.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-intel-haswell-1712073321.tar.gz` size: 0 MiB (2408 bytes) entries: 1 modules under 2023.06/software/linux/x86_64/intel/haswell/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/intel/haswell/software no software packages in tarball other under 2023.06/software/linux/x86_64/intel/haswell `.lmod/SitePackage.lua`
Apr 02 16:10:59 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8919.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
Apr 02 17:25:58 UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-intel-haswell-1712073321.tar.gz` to S3 bucket succeeded

eessi-bot · 2024-04-02T15:49:27Z

New job on instance eessi-bot-mc-aws for architecture x86_64-intel-skylake_avx512 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.04/pr_525/8920

date	job status	comment
Apr 02 15:49:26 UTC 2024	submitted	job id `8920` awaits release by job manager
Apr 02 15:49:54 UTC 2024	released	job awaits launch by Slurm scheduler
Apr 02 15:55:19 UTC 2024	running	job `8920` is running
Apr 02 16:08:50 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8920.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-1712073320.tar.gz` size: 0 MiB (2410 bytes) entries: 1 modules under 2023.06/software/linux/x86_64/intel/skylake_avx512/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/intel/skylake_avx512/software no software packages in tarball other under 2023.06/software/linux/x86_64/intel/skylake_avx512 `.lmod/SitePackage.lua`
Apr 02 16:08:50 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8920.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
Apr 02 17:26:17 UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-intel-skylake_avx512-1712073320.tar.gz` to S3 bucket succeeded

eessi-bot · 2024-04-02T15:49:31Z

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.04/pr_525/8921

date	job status	comment
Apr 02 15:49:30 UTC 2024	submitted	job id `8921` awaits release by job manager
Apr 02 15:49:46 UTC 2024	released	job awaits launch by Slurm scheduler
Apr 02 15:55:11 UTC 2024	running	job `8921` is running
Apr 02 16:13:04 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8921.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1712073287.tar.gz` size: 0 MiB (2398 bytes) entries: 1 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `.lmod/SitePackage.lua`
Apr 02 16:13:04 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8921.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
Apr 02 17:25:19 UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-amd-zen2-1712073287.tar.gz` to S3 bucket succeeded

eessi-bot · 2024-04-02T15:49:35Z

New job on instance eessi-bot-mc-aws for architecture x86_64-amd-zen3 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.04/pr_525/8922

date	job status	comment
Apr 02 15:49:34 UTC 2024	submitted	job id `8922` awaits release by job manager
Apr 02 15:49:48 UTC 2024	released	job awaits launch by Slurm scheduler
Apr 02 15:55:14 UTC 2024	running	job `8922` is running
Apr 02 16:06:37 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8922.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen3-1712073285.tar.gz` size: 0 MiB (2401 bytes) entries: 1 modules under 2023.06/software/linux/x86_64/amd/zen3/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen3/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen3 `.lmod/SitePackage.lua`
Apr 02 16:06:37 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8922.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
Apr 02 17:25:38 UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-amd-zen3-1712073285.tar.gz` to S3 bucket succeeded

eessi-bot · 2024-04-02T15:49:38Z

New job on instance eessi-bot-mc-aws for architecture aarch64-generic for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.04/pr_525/8923

date	job status	comment
Apr 02 15:49:37 UTC 2024	submitted	job id `8923` awaits release by job manager
Apr 02 15:49:43 UTC 2024	released	job awaits launch by Slurm scheduler
Apr 02 15:54:01 UTC 2024	running	job `8923` is running
Apr 02 16:03:17 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8923.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-generic-1712073233.tar.gz` size: 0 MiB (2409 bytes) entries: 1 modules under 2023.06/software/linux/aarch64/generic/modules/all no module files in tarball software under 2023.06/software/linux/aarch64/generic/software no software packages in tarball other under 2023.06/software/linux/aarch64/generic `.lmod/SitePackage.lua`
Apr 02 16:03:17 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8923.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
Apr 02 17:24:02 UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-aarch64-generic-1712073233.tar.gz` to S3 bucket succeeded

eessi-bot · 2024-04-02T15:49:42Z

New job on instance eessi-bot-mc-aws for architecture aarch64-neoverse_n1 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.04/pr_525/8924

date	job status	comment
Apr 02 15:49:41 UTC 2024	submitted	job id `8924` awaits release by job manager
Apr 02 15:49:45 UTC 2024	released	job awaits launch by Slurm scheduler
Apr 02 15:54:03 UTC 2024	running	job `8924` is running
Apr 02 16:04:24 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8924.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-neoverse_n1-1712073234.tar.gz` size: 0 MiB (2412 bytes) entries: 1 modules under 2023.06/software/linux/aarch64/neoverse_n1/modules/all no module files in tarball software under 2023.06/software/linux/aarch64/neoverse_n1/software no software packages in tarball other under 2023.06/software/linux/aarch64/neoverse_n1 `.lmod/SitePackage.lua`
Apr 02 16:04:24 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8924.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
Apr 02 17:24:21 UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-aarch64-neoverse_n1-1712073234.tar.gz` to S3 bucket succeeded

eessi-bot · 2024-04-02T15:49:45Z

New job on instance eessi-bot-mc-aws for architecture aarch64-neoverse_v1 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.04/pr_525/8925

date	job status	comment
Apr 02 15:49:44 UTC 2024	submitted	job id `8925` awaits release by job manager
Apr 02 15:50:56 UTC 2024	released	job awaits launch by Slurm scheduler
Apr 02 15:57:30 UTC 2024	running	job `8925` is running
Apr 02 16:04:26 UTC 2024	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-8925.out` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-aarch64-neoverse_v1-1712073408.tar.gz` size: 0 MiB (2413 bytes) entries: 1 modules under 2023.06/software/linux/aarch64/neoverse_v1/modules/all no module files in tarball software under 2023.06/software/linux/aarch64/neoverse_v1/software no software packages in tarball other under 2023.06/software/linux/aarch64/neoverse_v1 `.lmod/SitePackage.lua`
Apr 02 16:04:26 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ PASSED ] Ran 9/9 test case(s) from 9 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-8925.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
Apr 02 17:24:40 UTC 2024	uploaded	transfer of `eessi-2023.06-software-linux-aarch64-neoverse_v1-1712073408.tar.gz` to S3 bucket succeeded

…hen running in compat layer env

… a bind-mounted location

boegel

lgtm

Caspar van Leeuwen and others added 2 commits March 29, 2024 15:12

Make sure EESSI's SitePackage.lua loads host's SitePackage.lua's if t…

6e806fa

…hey exist

Forgot 'then'

4bc0369

boegel added the 2023.06-software.eessi.io 2023.06 version of software.eessi.io label Apr 2, 2024

boegel reviewed Apr 2, 2024

View reviewed changes

create_lmodsitepackage.py Outdated Show resolved Hide resolved

boegel requested changes Apr 2, 2024

View reviewed changes

create_lmodsitepackage.py Outdated Show resolved Hide resolved

Caspar van Leeuwen added 2 commits April 2, 2024 17:19

Changed isDir to isFile

b59d327

Account for the possibility that EESSI_PREFIX isn't set

9ff3cfa

boegel requested changes Apr 2, 2024

View reviewed changes

create_lmodsitepackage.py Outdated Show resolved Hide resolved

Corrected function name

f25bfe0

boegel previously approved these changes Apr 2, 2024

View reviewed changes

also test create_lmodsitepackage.py in CI

12d17a4

boegel dismissed their stale review via 12d17a4 April 2, 2024 16:14

Caspar van Leeuwen added 3 commits April 2, 2024 18:38

Make sure export is done inside test script, as environment is lost w…

60c46ce

…hen running in compat layer env

Change expected output - we need to account for the fact that this is…

d59238b

… a bind-mounted location

Output variable was undefined

1e8eb6e

boegel added the bot:deploy Ask bot to deploy missing software installations to EESSI label Apr 2, 2024

boegel approved these changes Apr 2, 2024

View reviewed changes

boegel merged commit 1eeb8ab into EESSI:2023.06-software.eessi.io Apr 2, 2024

trz42 mentioned this pull request Apr 5, 2024

Syncing code-base and CVMFS repository with EESSI (April 2024) NorESSI/software-layer#294

Open

51 tasks

casparvl mentioned this pull request Jun 4, 2024

Host injections, lmod hooks and moving gpu-related host-injection instructions EESSI/docs#188

Merged

casparvl deleted the allow_lmod_hook_host_injections branch August 15, 2024 15:03

Allow defining lmod hooks in host injections #525

Allow defining lmod hooks in host injections #525

Uh oh!

Conversation

casparvl commented Mar 29, 2024

Uh oh!

eessi-bot bot commented Mar 29, 2024

Uh oh!

casparvl commented Mar 29, 2024

Uh oh!

eessi-bot bot commented Mar 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Mar 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Mar 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Mar 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Mar 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Mar 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Mar 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Mar 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Mar 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

boegel commented Apr 2, 2024

Uh oh!

eessi-bot bot commented Apr 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Apr 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Apr 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Apr 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Apr 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Apr 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Apr 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Apr 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot bot commented Apr 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegel left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

eessi-bot bot commented Mar 29, 2024 •

edited

Loading

eessi-bot bot commented Mar 29, 2024 •

edited

Loading

eessi-bot bot commented Mar 29, 2024 •

edited

Loading

eessi-bot bot commented Mar 29, 2024 •

edited

Loading

eessi-bot bot commented Mar 29, 2024 •

edited

Loading

eessi-bot bot commented Mar 29, 2024 •

edited

Loading

eessi-bot bot commented Mar 29, 2024 •

edited

Loading

eessi-bot bot commented Mar 29, 2024 •

edited

Loading

eessi-bot bot commented Mar 29, 2024 •

edited

Loading

eessi-bot bot commented Apr 2, 2024 •

edited

Loading

eessi-bot bot commented Apr 2, 2024 •

edited

Loading

eessi-bot bot commented Apr 2, 2024 •

edited

Loading

eessi-bot bot commented Apr 2, 2024 •

edited

Loading

eessi-bot bot commented Apr 2, 2024 •

edited

Loading

eessi-bot bot commented Apr 2, 2024 •

edited

Loading

eessi-bot bot commented Apr 2, 2024 •

edited

Loading

eessi-bot bot commented Apr 2, 2024 •

edited

Loading

eessi-bot bot commented Apr 2, 2024 •

edited

Loading