Since GNULD_LINKER_IS_BFD, incremental SOF build fails with a very confusing "-fuse-ld=bfd: unknown flag"

[marc-hb]

Describe the bug

Commit 04fd862 "linker: ld: GNULD_LINKER_IS_BFD to indicate if ld.bfd is used" causes extremely confusing SOF build failures.

To Reproduce
Steps to reproduce the behavior:

git checkout 04fd862f68ba8

ZEPHYR_TOOLCHAIN_VARIANT=xcc
XTENSA_TOOLCHAIN_PATH=$HOME/XCC/install/tools
TOOLCHAIN_VER=RG-2017.8-linux
XTENSA_SYSTEM=$HOME/XCC/install/builds/RG-2017.8-linux/cavs2x_LX6HiFi3_2017_8/config

export XTENSA_SYSTEM XTENSA_TOOLCHAIN_PATH ZEPHYR_TOOLCHAIN_VARIANT TOOLCHAIN_VER

rm -rf build

west build -b intel_adsp_cavs25 modules/audio/sof/app
west build -b intel_adsp_cavs25 modules/audio/sof/app -- -DFOO=bar

Since commit 04fd862, the second, incremental west build command now fails with the very confusing error $HOME/XCC/install/tools/RG-2017.8-linux/XtensaTools/bin/xt-xcc ERROR parsing -fuse-ld=bfd: unknown flag

No failure with the Zephyr SDK.

Expected behavior
At least a vaguely useful error message. Since #40431 (comment) I'm aware you're not supposed to have any CMake parameters on incremental builds but in this case the experience is really miserable: it took me hours to isolate this problem. Also: why does it fail with one toolchain but not with another? Confusingly inconsistent.

Impact
The incremental SOF build fails and users don't have a clue why.

Workaround: build from scratch with -p. Especially slow due to the license server.

Environment (please complete the following information):

• OS: Linux
• Toolchain xt-xcc
• Any commit after 04fd862

[marc-hb]

it took me hours to isolate this problem.

One of the reasons is because there is no -DFOO-bar in the real-world case. There is an -DOVERLAY_CONFIG=debug.conf instead, itself indirectly coming from -d option.

[marc-hb]

https://docs.zephyrproject.org/latest/develop/west/build-flash-debug.html#one-time-cmake-arguments states:

Passing additional CMake arguments like this forces west build to re-run CMake, even if a build system has already been generated.

This does not seem true in this example. If CMake were re-run then the build would just be slower, it would not fail.

[dcpleung]

Could you try adding CACHE to set(GNULD_LINKER_IS_BFD TRUE)? (==> set(GNULD_LINKER_IS_BFD TRUE CACHE) and for the FALSE one too? I don't have the RG xcc setup to test this at the moment, so it probably is faster if you can try it out.

[marc-hb]

Could you try adding CACHE to set(GNULD_LINKER_IS_BFD ...

This seems to work! I'm impressed :-) I'm not 100% sure what gets built and when but I can't make the build fail anymore with this CACHE change.

BTW git grep -C 1 set.*CACHE.*BOOL shows ON/OFF, not TRUE/FALSE.

https://docs.zephyrproject.org/latest/develop/west/build-flash-debug.html#one-time-cmake-arguments states:

Passing additional CMake arguments like this forces west build to re-run CMake, even if a build system has already been generated.

This does not seem true in this example.

Correcting myself: this is not true in this example because CMake does not care about -DFOO=bar. HOWEVER when flipping something "real" like -DCONFIG_DEBUG=x then the build system does notice and does re-run CMake when the value has changed. So maybe passing CMake args on incremental builds is not so bad after all?