build failure for Python 3.13 in EESSI 2025.06 due to missing `__gcov_indirect_call` for `test_embed` during profile-guided optimization

[boegel]

The build of Python-3.13.1-GCCcore-14.2.0.eb for EESSI 2025.06 is failing with:

LD_LIBRARY_PATH=/tmp/eessibot/easybuild/build/Python/3.13.1/GCCcore-14.2.0/Python-3.13.1 ./python -m test --pgo --timeout=
...
0:00:13 load avg: 1.33 [16/44] test_embed
test test_embed failed
0:00:14 load avg: 1.33 [17/44] test_float -- test_embed failed (71 failures)

The actual problem becomes clear when trying to run the testembed binary from the interactive debug shell that can be started with the cmd.sh script that is generated by EasyBuild:

eb-shell> Programs/_testembed
Programs/_testembed: symbol lookup error: Programs/_testembed: undefined symbol: __gcov_indirect_call

Indeed, the _testembed binary is missing this symbol:

eb-shell> nm Programs/_testembed | grep __gcov_indirect_call
                 U __gcov_indirect_call
                 U __gcov_indirect_call_profiler_v4

(U stands for "undefined" here)

This symbol is supposed to get resolved by the libgcov.a static library provided by GCC:

eb-shell> nm $EBROOTGCCCORE/lib/gcc/x86_64-pc-linux-gnu/14.2.0/libgcov.a | grep __gcov_indirect_call
0000000000000000 B __gcov_indirect_call
0000000000000000 T __gcov_indirect_call_profiler_v4
0000000000000220 T __gcov_indirect_call_profiler_v4_atomic

It was a bit tricky to figure out, but re-running the command that links the _testembed binary with an extra option (--trace-symbol=__gcov_indirect_call) to let ld report how the __gcov_indirect_call symbol gets resolved helps to pinpoint the problem:

$ gcc -Wl,--trace-symbol=__gcov_indirect_call ... -fprofile-generate ... -o Programs/_testembed: ...
/cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/usr/bin/ld: Programs/_testembed.o (symbol from plugin): reference to __gcov_indirect_call
/cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/usr/bin/ld: ./libpython3.13.so: definition of __gcov_indirect_call
/cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/usr/bin/ld: /tmp/eb-fpnz_fhe/ccGRDgYP.lto.o: reference to __gcov_indirect_call

This shows that the __gcov_indirect_call is actually being resolved through the libpython3.13.so in the current directory.
Indeed, this (temporary, profile-enabled) library provides this symbol:

$ nm ./libpython3.13.so | grep __gcov_indirect_call
0000000000000020 B __gcov_indirect_call
000000000063f320 T __gcov_indirect_call_profiler_v4
000000000063f540 T __gcov_indirect_call_profiler_v4_atomic

Checking which libraries are used by _testembed at runtime then reveals the problem:

eb-shell> ldd Programs/_testembed
        linux-vdso.so.1 (0x000014f8f1e31000)
        libpython3.13.so.1.0 => /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/usr/lib/../lib64/libpython3.13.so.1.0 (0x000014f8f1800000)
        libm.so.6 => /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/lib/../lib64/libm.so.6 (0x000014f8f1714000)
        libc.so.6 => /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/lib/../lib64/libc.so.6 (0x000014f8f1539000)
        /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 (0x000014f8f1e33000)

Because of the RPATH section in _testembed, which includes the path to libraries provided by the compat layer, the libpython3.13.so.1.0 being used at runtime is the one from the compat layer.
And this one does not provide the __gcov_indirect_call symbol, because it wasn't built with -fprofile-generate (which implies linking in -lgcov):

eb-shell> nm /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/usr/lib/../lib64/libpython3.13.so.1.0
nm: /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/usr/lib/../lib64/libpython3.13.so.1.0: no symbols

Hence, the LD_LIBRARY_PATH=... that is used as prefix for the ./python -m test --pgo --timeout= command doesn't have the intended effect anymore of making sure that libpython3.13.so is being picked up from the current working directory, because it gets overruled by the RPATH section in the _testembed binary.
It's also a stroke of "bad luck", since there happens to be a libpython3.13.so.1.0 in the compat layer. If not, then eventually the path specified in $LD_LIBRARY_PATH would be picked up, and it would all work out fine. This explains why this problem doesn't pop up with Python-3.12.3-GCCcore-13.3.0.eb.

[ocaisa]

Given the situation here, can we just sed s/LD_LIBRARY_PATH/LD_PRELOAD/ on the right file and move on?

[boegel]

Some more context here: this is really an unfortunate "perfect storm" kind of situation...
After exploring some options to fix this, I have a better view on things now...

The Python build scripts set $LD_LIBRARY_PATH with the intention to pick up the libpython3.13.so from the build directory at runtime, but they also make sure that -L. is used to pick up on that library at build time.
As the output of gcc -Wl,--trace-symbol=... above show, that's also working as expected (at build time).

However, our RPATH wrappers don't inject the current working directory into the RPATH section, for two reasons:

• it's specified as a relative path (.);
• even if it would be an absolute path (say using -L ${PWD}), it still wouldn't be used because the RPATH wrapper script filters out paths that correspond to the build directory (since that will be removed after the installation is completed);

We could come up with an "allowlist" mechanism for the RPATH wrappers so the Python easyblock can specify that the build directory should not be filtered out, but since this is the only case where this problem has popped up I'm not sure that's worth the effort right now...

Given the situation here, can we just sed s/LD_LIBRARY_PATH/LD_PRELOAD/ on the right file and move on?

That seems like a good option to try indeed (doing that next), and that change in the Python build scripts could perhaps even be pushed upstream, since the $LD_LIBRARY_PATH approach is a bit brittle.

[trz42]

Yeah, LD_PRELOAD is the only option here.

The test program looks perfectly built (correct RPATH), but we are hit by having a library with the same name in the compat layer (could be in any other directory in the RPATH) and the directory that will contain the library does not exist yet because the installation is still in the build step. Plus LD_LIBRARY_PATH does not overwrite RPATH.

[boegel]

Working patch file for Python 3.13.1:

$ cat Python-3.13.1_runshared-ld-preload.patch
diff -ru Python-3.13.1.orig/configure Python-3.13.1/configure
--- Python-3.13.1.orig/configure        2024-12-03 18:59:52.000000000 +0100
+++ Python-3.13.1/configure     2025-10-10 08:11:07.387655393 +0200
@@ -7586,7 +7586,8 @@
     Linux*|GNU*|NetBSD*|FreeBSD*|DragonFly*|OpenBSD*|VxWorks*)
       LDLIBRARY='libpython$(LDVERSION).so'
       BLDLIBRARY='-L. -lpython$(LDVERSION)'
-      RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
+      # make sure that right libpython.so is used, i.e. the one from current (build) directory
+      RUNSHARED=LD_PRELOAD=`pwd`/$LDLIBRARY${LD_PRELOAD:+:${LD_PRELOAD}}

       # The Android Gradle plugin will only package libraries whose names end
       # with ".so".

This looks safe to always apply, so I'll open an easyconfigs PR for this one for recent Python versions...

[boegel]

The same problem with the failing test_embed actually also happened with the build of `` in EESSI 2023.06 (since libpython3.11.so is part of the compat layer):

$ bzgrep 'test_embed.*fail' /cvmfs/software.eessi.io/versions/202*.06/software/linux/x86_64/amd/zen2/software/Python/3.11.3-GCCcore-12.3.0/easybuild/*log*
test test_embed failed
0:00:37 load avg: 2.58 [17/44] test_float -- test_embed failed (65 failures)
        test test_embed failed
        0:00:37 load avg: 2.58 [17/44] test_float -- test_embed failed (65 failures)

In Python 3.11, failing tests were still being ignored though when running python -m test --pgo, see https://github.com/python/cpython/blob/v3.11.3/Makefile.pre.in#L640

edit: This was changed in Python 3.13, see python/cpython#110295

[boegel]

• add patch for Python 3.13 to fix test failure that may occur when collecting performance profile for profile-guided optimization (PGO) easybuilders/easybuild-easyconfigs#24189

[boegel]

• {2025.06}[2025a] Python 3.13.1 #1232