Issues lurking with `gobject-introspection`

[ocaisa]

With something that was using Meson I was seeing

/bin/bash: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /cvmfs/pilot.eessi-hpc.org/versions/2021.12/compat/linux/x86_64/usr/lib64/libtinfo
.so.6)

which I'm guessing meant it was picking the host ld and not the EasyBuild wrapper. Looking at the configure step I saw

C compiler for the host machine: gcc (gcc 10.3.0 "gcc (GCC) 10.3.0")
C linker for the host machine: gcc ld.gold 2.37

which would imply it is using the ld.gold used by gcc (which seems right since it reports 2.37, the version in the prefix, not 2.30 the version in the host).

Anyway, it seemed like a good idea to set the linker directly (as per the docs) with

preconfigopts = "export CC_LD=$(which ld) &&"

but actually this throws an error

== 2022-11-08 12:59:53,619 build_log.py:169 ERROR EasyBuild crashed with an error (at easybuild/software/EasyBuild/4.6.0/lib/python3.9/site-packages/easybuild/base/exceptions.py:124 in __init__): cmd "export CC_LD=$(which ld) && meson --prefix /project/def-sponsor00/easybuild/software/at-spi2-core/2.40.2-GCCcore-10.3.0 --libdir lib  /project/60005/easybuild/build/atspi2core/2.40.2/GCCcore-10.3.0/at-spi2-core-2.40.2/" exited with exit code 1 and output:
The Meson build system
Version: 0.58.0
Source dir: /project/60005/easybuild/build/atspi2core/2.40.2/GCCcore-10.3.0/at-spi2-core-2.40.2
Build dir: /project/60005/easybuild/build/atspi2core/2.40.2/GCCcore-10.3.0/easybuild_obj
Build type: native build
Project name: at-spi2-core
Project version: 2.40.2

../at-spi2-core-2.40.2/meson.build:1:0: ERROR: Unsupported linker, only bfd, gold, and lld are supported, not /tmp/eb-nf5k_pyz/tmp1a3n724j/rpath_wrappers/ld_wrapper/ld.

[ocaisa]

Urgh, this may not be an easy fix mesonbuild/meson#6662

[ocaisa]

I don't see how we can give a full path to a linker: https://github.com/mesonbuild/meson/blob/master/mesonbuild/compilers/mixins/gnu.py#L311-L316

[ocaisa]

Ah, you are meant to set the linker flavour with the environment variables...not sure if that means the problem is something else or not

[ocaisa]

This may be a false flag, I think introspection is up to something, disabling that with

meson --prefix /project/def-sponsor00/easybuild/software/at-spi2-core/2.40.2-GCCcore-10.3.0 -Dintrospection=no --libdir lib

allowed the package to install

[ocaisa]

It is worth noting though that EasyBuild does not currently create compiler wrappers for ld.bfd and ld.gold

[eli-schwartz]

Ah, you are meant to set the linker flavour with the environment variables...not sure if that means the problem is something else or not

The environment variable tries to map -fuse-ld values, which expects to see a flavor.

If you want to set a linker that GCC doesn't specifically know about, you probably want gcc -B/path/to/ld_wrapper/ (set via LDFLAGS).

[ocaisa]

The full error with gobject-introspection is

[49/62] Generating Atspi-2.0.gir with a custom command
FAILED: atspi/Atspi-2.0.gir 
/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/bin/g-ir-scanner --no-libtool --namespace=Atspi --nsversion=2.0 --warn-all --output atspi/Atspi-2.0.gir --c-include=atspi/atspi.h --pkg-export=atspi-2 --quiet -I/project/60005/easybuild/build/atspi2core/2.40.2/GCCcore-10.3.0/at-spi2-core-2.40.2/atspi -I/project/60005/easybuild/build/atspi2core/2.40.2/GCCcore-10.3.0/easybuild_obj/atspi -I./. -I../at-spi2-core-2.40.2/. -I./registryd -I../at-spi2-core-2.40.2/registryd -I./. -I../at-spi2-core-2.40.2/. --filelist=/project/60005/easybuild/build/atspi2core/2.40.2/GCCcore-10.3.0/easybuild_obj/atspi/libatspi.so.0.0.1.p/Atspi_2.0_gir_filelist --include=DBus-1.0 --include=GLib-2.0 --include=GObject-2.0 --symbol-prefix=atspi --identifier-prefix=Atspi --cflags-begin -D_POSIX_C_SOURCE=200809L -D_DEFAULT_SOURCE -I./. -I../at-spi2-core-2.40.2/. -I./registryd -I../at-spi2-core-2.40.2/registryd -I/cvmfs/pilot.eessi-hpc.org/versions/2021.12/compat/linux/x86_64/usr/include/dbus-1.0 -I/cvmfs/pilot.eessi-hpc.org/versions/2021.12/compat/linux/x86_64/usr/lib64/dbus-1.0/include -I/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/include/glib-2.0 -I/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib/glib-2.0/include -I/cvmfs/pilot.eessi-hpc.org/versions/2021.12/compat/linux/x86_64/usr/include '-DG_LOG_DOMAIN="dbind"' -I/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/include/gobject-introspection-1.0 -I/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/include -I/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/include -I/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/include --cflags-end -I./. -I../at-spi2-core-2.40.2/. --add-include-path=/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/share/gir-1.0 --add-include-path=./. --add-include-path=../at-spi2-core-2.40.2/. -L./atspi --library atspi -L/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib64 -L/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib -L/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib64 -L/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib -L/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib64 -L/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib -L/project/def-sponsor00/easybuild/software/Meson/0.58.0-GCCcore-10.3.0/lib64 -L/project/def-sponsor00/easybuild/software/Meson/0.58.0-GCCcore-10.3.0/lib -L/cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/GCCcore/10.3.0/lib64 -L/cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/GCCcore/10.3.0/lib -L/cvmfs/pilot.eessi-hpc.org/versions/2021.12/compat/linux/x86_64/usr/lib64 --extra-library=dbus-1 -L/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib --extra-library=gobject-2.0 --extra-library=glib-2.0 -L/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib --extra-library=X11 --extra-library=Xtst --extra-library=Xi -L/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib --extra-library=girepository-1.0 --sources-top-dirs /project/60005/easybuild/build/atspi2core/2.40.2/GCCcore-10.3.0/at-spi2-core-2.40.2/subprojects/ --sources-top-dirs /project/60005/easybuild/build/atspi2core/2.40.2/GCCcore-10.3.0/easybuild_obj/subprojects/
/bin/bash: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /cvmfs/pilot.eessi-hpc.org/versions/2021.12/compat/linux/x86_64/usr/lib64/libtinfo.so.6)
linking of temporary binary failed: Command '['gcc', '-o', '/project/60005/easybuild/build/atspi2core/2.40.2/GCCcore-10.3.0/easybuild_obj/tmp-introspect07cjsv_q/Atspi-2.0', '-I/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/include', '-I/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/include', '-I/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/include', '-O2', '-ftree-vectorize', '-march=native', '-fno-math-errno', '/project/60005/easybuild/build/atspi2core/2.40.2/GCCcore-10.3.0/easybuild_obj/tmp-introspect07cjsv_q/Atspi-2.0.o', '-L.', '-Wl,-rpath,.', '-Wl,--no-as-needed', '-L./atspi', '-L/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib64', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib64', '-L/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib', '-L/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib64', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib64', '-L/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib', '-L/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib64', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib64', '-L/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib', '-L/project/def-sponsor00/easybuild/software/Meson/0.58.0-GCCcore-10.3.0/lib64', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/Meson/0.58.0-GCCcore-10.3.0/lib64', '-L/project/def-sponsor00/easybuild/software/Meson/0.58.0-GCCcore-10.3.0/lib', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/Meson/0.58.0-GCCcore-10.3.0/lib', '-L/cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/GCCcore/10.3.0/lib64', '-Wl,-rpath,/cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/GCCcore/10.3.0/lib64', '-L/cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/GCCcore/10.3.0/lib', '-Wl,-rpath,/cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/GCCcore/10.3.0/lib', '-L/cvmfs/pilot.eessi-hpc.org/versions/2021.12/compat/linux/x86_64/usr/lib64', '-Wl,-rpath,/cvmfs/pilot.eessi-hpc.org/versions/2021.12/compat/linux/x86_64/usr/lib64', '-L/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib', '-L/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib', '-L/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib', '-Wl,-rpath,/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib', '-latspi', '-ldbus-1', '-lgobject-2.0', '-lglib-2.0', '-lX11', '-lXtst', '-lXi', '-lgirepository-1.0', '-L/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib', '-lgio-2.0', '-lgobject-2.0', '-Wl,--export-dynamic', '-lgmodule-2.0', '-pthread', '-lglib-2.0', '-L/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib64', '-L/project/def-sponsor00/easybuild/software/X11/20210518-GCCcore-10.3.0/lib', '-L/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib64', '-L/project/def-sponsor00/easybuild/software/GLib/2.68.2-GCCcore-10.3.0/lib', '-L/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib64', '-L/project/def-sponsor00/easybuild/software/GObject-Introspection/1.68.0-GCCcore-10.3.0/lib', '-L/project/def-sponsor00/easybuild/software/Meson/0.58.0-GCCcore-10.3.0/lib64', '-L/project/def-sponsor00/easybuild/software/Meson/0.58.0-GCCcore-10.3.0/lib', '-L/cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/GCCcore/10.3.0/lib64', '-L/cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/software/GCCcore/10.3.0/lib']' returned non-zero exit status 1.

specifically, we probably need to take a look inside the python script g-ir-scanner and see where things are going wrong.

[ocaisa]

I think this has something to do with the use of /bin/bash in EasyBuild which is allowing us to escape the prefix environment (and means we start to use the system linker)

[boegel]

@ocaisa Can you check if you see the same problem when using python3 -O -m easybuild.main ... instead of eb ...?

[ocaisa]

@boegel It didn't help, and I checked the bash shell at the time the build command is called:

== 2022-12-02 15:31:57,205 build_log.py:169 ERROR EasyBuild crashed with an error (at easybuild/software/EasyBuild/4.6.0/lib/python3.9/site-packages/easyb
uild/base/exceptions.py:124 in __init__): cmd "echo $0 && ninja -j 2 " exited with exit code 1 and output:
/cvmfs/pilot.eessi-hpc.org/versions/2021.12/compat/linux/x86_64/bin/bash
...

so I'm not sure where exactly things go wrong.

[boegel]

see also #428 (comment), which suggests that ldd is the culprit here

As discussing during today's support meeting, we should check whether the hardcoded /bin/bash in ldd is deliberate; it seems like a bug to me in Gentoo Prefix?

The problem here seems to be that LD_LIBRARY_PATH is being set to the location of libraries in the compat layer, which breaks the host's /bin/bash...

Any ideas here @amadio?

Possible workaround could be:

• runtime patch to replace call to ldd to inject unset LD_LIBRARY_PATH;
• call ldd via the bash of the compat layer (effectively skipping the hardcoded /bin/bash);