correct libgcc not getting linked for CONFIG_FLOAT=y on ARM

[zephyrbot]

Reported by Kumar Gala:

When enabling CONFIG_FLOAT=y we aren't linking in the correct libgcc.a on ARM architectures. There are a number of issues here:

1. In the Zephyr toolchain the Makefile (scripts/Makefile.toolchain.zephyr) we need to set CROSS_COMPILE_arm_isa to 'fpu' if CONFIG_FLOAT is enabled.
2. if we do #1, we run into a build/link issue:

/opt/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/libexec/arm-poky-eabi/gcc/arm-poky-eabi/5.2.0/real-ld: error: /opt/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/fpu/libgcc.a(_powisf2.o) uses VFP register arguments, .tmp_zephyr.prebuilt does not
/opt/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/libexec/arm-poky-eabi/gcc/arm-poky-eabi/5.2.0/real-ld: failed to merge target specific data of file /opt/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/fpu/libgcc.a(_powisf2.o)
/opt/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/libexec/arm-poky-eabi/gcc/arm-poky-eabi/5.2.0/real-ld: error: /opt/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/fpu/libgcc.a(_udivmoddi4.o) uses VFP register arguments, .tmp_zephyr.prebuilt does not
/opt/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/libexec/arm-poky-eabi/gcc/arm-poky-eabi/5.2.0/real-ld: failed to merge target specific data of file /opt/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/fpu/libgcc.a(_udivmoddi4.o)
collect2: error: ld returned 1 exit status

This is due to building the kernel with -mfloat-abi=softfp, while the Zephyr toolchain only supports -mfloat-abi=hard.

While we can quickly fixing this with some minor tweaks, are more proper fix would utilize -print-libgcc-file-name to get the libgcc path and we'd pickup:

https://gerrit.zephyrproject.org/r/#/c/2899/

To get multilib support for -mfloat-abi=softfp, and -mfloat-abi=hard

(Imported from Jira ZEP-555)

[zephyrbot]

by Juro Bystricky:

Yes, multilibs need some cleanup. For example, the following should be in Makefile.toolchain.zephyr:

{code:java}
ifeq ($(ARCH),arm)
CROSS_COMPILE_TARGET_arm = arm-poky-eabi
CROSS_COMPILE_arm=$(TOOLCHAIN_HOME)/usr/bin/$(CROSS_COMPILE_TARGET_arm)/$(CROSS_COMPILE_TARGET_arm)-
SYSROOT_arm := ${ZEPHYR_SDK_INSTALL_DIR}/sysroots/armv5-poky-eabi

libgcc

LIBGCC_DIR_arm = $(shell dirname $(CROSS_COMPILE_arm)gcc --sysroot=$(SYSROOT_arm) $(KBUILD_CFLAGS) -print-libgcc-file-name)
LIB_INCLUDE_DIR_arm += -L ${LIBGCC_DIR_arm}

newlib lib/include

NEWLIB_DIR_arm = $(shell $(CROSS_COMPILE_arm)gcc --sysroot=$(SYSROOT_arm) $(KBUILD_CFLAGS) -print-multi-directory)
LIB_INCLUDE_DIR_arm += -L $(SYSROOT_arm)/usr/lib/$(NEWLIB_DIR_arm)/
TOOLCHAIN_CFLAGS_arm = -I $(SYSROOT_arm)/usr/include
endif

{code}

Similar setup for other multilib toolchains (ARC, Nios2, ...). Basically, we query the compiler itself which library should be used given compiler flags (options).
One caveat: KBUILD_CFLAGS must be set, as it contains all GCC command line options.
Currently Makefile.toolchain.zephyr is included before this is done, so some re-ordering is needed as well.

[zephyrbot]

by Kumar Gala:

Fixed by combination of SDK 0.8.2 (which supports multilibs for softfp and hard fp ABIs). Plus change to allow user to select which Floating Point ABI they want to build with.

[zephyrbot]

by Sharron LIU:

Please reporter verify and close this. Thanks.

[zephyrbot]

by Mark Linkmeyer:

Hi Kumar Gala , as the Reporter of this bug we need you to verify it's fixed since it's in the Resolved state. Please verify it ASAP so we know whether or not more work is required to fix it before the upcoming 1.5 release. If it's fixed, please move it to the Verified state. Thanks!

[zephyrbot]

by Marcus Shawcroft:

The underlying issue is still present with the SDK 0.8.2, but manifests when building with hard float disabled.

The multilib configuration for arm-poki-eabi-gcc is:
.;
thumb;@mthumb
fpu;@mfloat-abi=hard
thumb/thumb2;@mthumb@march=armv7

The KBUILD_CFLAGS options passed for an frdm_k64f build include:
-mcpu=cortex-m4 -mthumb -march=armv7e-m

The architecture 'armv7e-m' is correct for a cortex-m4. This set of options will match the first multlib (ie thumb;@thumb). The libgcc.a file in that directory was built for armv4t (where thumb and arm state are legitimate). The armv7e-m architecture is thumb2 only, hence a binary linked against this first multilib is illegal.

For reference, there is comprehensive multilib support for the full range of ARM R and M profile architecture permutations provided in the the ARM/embedded-5 upstream GCC branch. See here for the first commit:
https://gcc.gnu.org/viewcvs/gcc/branches/ARM/embedded-5-branch/gcc/config/arm/t-baremetal?revision=231750&view=markup

This patch provides support for the following multilibs:
.;
thumb;@mthumb
fpu;@mfloat-abi=hard
armv6-m;@mthumb@march=armv6s-m
armv7-m;@mthumb@march=armv7-m
armv7e-m;@mthumb@march=armv7e-m
armv7-ar/thumb;@mthumb@march=armv7
armv8-m.base;@mthumb@march=armv8-m.base
armv8-m.main;@mthumb@march=armv8-m.main
armv7e-m/softfp/fpv5-sp-d16;@mthumb@march=armv7e-m@mfloat-abi=softfp@mfpu=fpv5-sp-d16
armv7e-m/softfp/fpv5-d16;@mthumb@march=armv7e-m@mfloat-abi=softfp@mfpu=fpv5-d16
armv7e-m/softfp;@mthumb@march=armv7e-m@mfloat-abi=softfp@mfpu=fpv4-sp-d16
armv7e-m/fpu/fpv5-sp-d16;@mthumb@march=armv7e-m@mfloat-abi=hard@mfpu=fpv5-sp-d16
armv7e-m/fpu/fpv5-d16;@mthumb@march=armv7e-m@mfloat-abi=hard@mfpu=fpv5-d16
armv7e-m/fpu;@mthumb@march=armv7e-m@mfloat-abi=hard@mfpu=fpv4-sp-d16
armv7-ar/thumb/softfp;@mthumb@march=armv7@mfloat-abi=softfp@mfpu=vfpv3-d16
armv7-ar/thumb/fpu;@mthumb@march=armv7@mfloat-abi=hard@mfpu=vfpv3-d16
armv8-m.main/softfp/fpv5-sp-d16;@mthumb@march=armv8-m.main@mfloat-abi=softfp@mfpu=fpv5-sp-d16
armv8-m.main/softfp/fpv5-d16;@mthumb@march=armv8-m.main@mfloat-abi=softfp@mfpu=fpv5-d16
armv8-m.main/fpu/fpv5-sp-d16;@mthumb@march=armv8-m.main@mfloat-abi=hard@mfpu=fpv5-sp-d16
armv8-m.main/fpu/fpv5-d16;@mthumb@march=armv8-m.main@mfloat-abi=hard@mfpu=fpv5-d16

While this is not complete, it does provide a viable lowest common denominator for all current arm aarch32 cortex-m/r architectures.

This patch/support has not yet been presented for inclusion in upstream GCC trunk, but my understanding is that this is likely in the near future.

w.r.t the immediate issue in zephyr, possible short term options include:
-) Add multilibs to cover just the current issue of broken cortex-m4 / aarch32e-m
-) Backport the relevant multilib patches from upstream gcc/embedded-5 to the zephyr sdk (these patches are ARM centric and hence are not likely to destabalize the compiler behaviour on the other architecture we care about in zephyr).

It's worth noting that folks currently working on cortex-m0 / m0+ support are going to hit the same issue (cortex-m0 == armv6-m), hence a minimum viable set of armv6-m multlibs will be required to enable their work.

[zephyrbot]

by Marcus Shawcroft:

This is still broken:

with 0.8.2. On frdm_k64f configured CONFIG_FP_SOFTABI an inappropriate libgcc.a is linked resulting in ARM state code being executed on arm7e-m

with CONFIG_FP_HARDABI we get a link error (one object uses VFP registers another does not).

See my previous comment above for more detail.

[zephyrbot]

by Kumar Gala:

What error are you seeing. How to reproduce?

[zephyrbot]

by Marcus Shawcroft:

Reproduce like this:

$ cat prj.conf
CONFIG_STDOUT_CONSOLE=y
CONFIG_FLOAT=y

$ cat src/main.c
double f = 2.0;

int
main ()
{
printf ("hello\n");
f = f / 3;
printf ("f=%f\n", f);
printf ("world\n");
}

$ make BOARD=frdm_k64f

gives....
/home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-gcc -T linker.cmd @zephyr.lnk -o .tmp_zephyr.prebuilt
/home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/libexec/arm-poky-eabi/gcc/arm-poky-eabi/5.2.0/real-ld: error: .tmp_zephyr.prebuilt uses VFP register arguments, /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/libgcc.a(_udivmoddi4.o) does not
/home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/libexec/arm-poky-eabi/gcc/arm-poky-eabi/5.2.0/real-ld: failed to merge target specific data of file /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/libgcc.a(_udivmoddi4.o)

Adding CONFIG_FP_SOFTABI:

Results in an apparently successful build. Howver the binary produced contains ARM state code. If you try to execute the resulting binary it will repeatedly fault.

For direct evidence of the issue:
$ arm-none-eabi-objdump -d outdir/frdm_k64f/zephyr.elf | grep -A 10 aeabi_ddiv | head -n 10
00000958 <__aeabi_ddiv>:
958: e92d4070 push {r4, r5, r6, lr}
95c: e3a0c0ff mov ip, #255 ; 0xff
960: e38ccc07 orr ip, ip, #1792 ; 0x700
964: e01c4a21 ands r4, ip, r1, lsr #20
968: 101c5a23 andsne r5, ip, r3, lsr #20
96c: 1134000c teqne r4, ip
970: 1135000c teqne r5, ip
974: 0b00005e bleq af4 <__aeabi_ddiv+0x19c>
978: e0444005 sub r4, r4, r5

ie ARM state code.

[zephyrbot]

by Kumar Gala:

Not able to reproduce. I'm able to run with both CONFIG_FP_SOFTABI and CONFIG_FP_HARDABI without issues. I also don't see the compile issue with HARDABI. Can you build with V=1 and report the link lines you are seeing.

[zephyrbot]

by Marcus Shawcroft:

Its possible to demonstrate correct behaviour here using the pre-built toolchain from https://launchpad.net/gcc-arm-embedded

$ arm-none-eabi-gcc -v
...
gcc version 5.4.1 20160609 (release)
...

$ make BOARD=frdm_k64f CROSS_COMPILE=arm-none-eabi-

Generates binaries that execute correctly in both of the examples I gave above.

[zephyrbot]

by Marcus Shawcroft:

Hi, Link lines as request:

  rm -f libzephyr.a; /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-ar rcTD libzephyr.a  drivers/built-in.o    lib/built-in.o  kernel/built-in.o  misc/built-in.o  net/built-in.o  boards/built-in.o  ext/built-in.o  usb/built-in.o  fs/built-in.o  arch/built-in.o
/home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-gcc -x assembler-with-cpp -nostdinc -undef -E -P \
	 -D__GCC_LINKER_CMD__ -I/home/marcus/work/zephyr/zephyr-project/include \
	-I./include/generated  /home/marcus/work/zephyr/zephyr-project/arch/arm/soc/nxp_kinetis/k6x/linker.ld -o linker.cmd
  ( echo -nostartfiles -nodefaultlibs -nostdlib -static -Wl,-X -Wl,-N -Wl,--gc-sections -Wl,--build-id=none; echo "-Wl,-Map=zephyr.map"; echo "-L ./include/generated"; echo "-u _OffsetAbsSyms -u _ConfigAbsSyms"; echo "-e __start"; echo "-Wl,--start-group"; echo "-Wl,--whole-archive"; echo ""; echo " src/built-in.o"; echo "libzephyr.a"; echo "-Wl,--no-whole-archive"; echo "./arch/arm/core/offsets/offsets.o"; echo "-Wl,--end-group"; echo "-L /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/thumb -L /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/thumb  -lgcc"; ) > zephyr.lnk
/home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-gcc -T linker.cmd @zephyr.lnk -o .tmp_zephyr.prebuilt
  ( /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-objdump -S zephyr.elf > zephyr.lst; /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-objcopy -S -O binary -R .note -R .comment -R COMMON -R .eh_frame zephyr.elf zephyr.bin; /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-strip -s -o zephyr.strip zephyr.elf; )

[zephyrbot]

by Marcus Shawcroft:

Kumar, when run, does the example above print 0.0000 or something resembling 0.6 ?

[zephyrbot]

by Kumar Gala:

I saw 0.0000 once, but than starting seeing 0.6.

Can you try removing '-mfpu=fpv4-sp-d16' from arch/arm/Makefile and see what you get. I'm noticing that -mfloat-abi=hard -mfpu=fpv4-sp-d16 isn't getting me the code I'd expect (with either the SDK toolchain or the ARM embedded toolchain). However if I drop the -mfpu option things seem more inline with what I'd expect.

[zephyrbot]

by Marcus Shawcroft:

Configured for CONFIG_FP_HARDABI with '-mfpu=fpv4-sp-d16' from arch/arm/Makefile the link fails with:

   /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-ld -nostartfiles -nodefaultlibs -nostdlib -static   -r -o src/built-in.o src/main.o 
/home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-gcc -x assembler-with-cpp -nostdinc -undef -E -P \
	 -D__GCC_LINKER_CMD__ -I/home/marcus/work/zephyr/zephyr-project/include \
	-I./include/generated  /home/marcus/work/zephyr/zephyr-project/arch/arm/soc/nxp_kinetis/k6x/linker.ld -o linker.cmd
  rm -f libzephyr.a; /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-ar rcTD libzephyr.a  drivers/built-in.o    lib/built-in.o  kernel/built-in.o  misc/built-in.o  net/built-in.o  boards/built-in.o  ext/built-in.o  usb/built-in.o  fs/built-in.o  arch/built-in.o
  ( echo -nostartfiles -nodefaultlibs -nostdlib -static -Wl,-X -Wl,-N -Wl,--gc-sections -Wl,--build-id=none; echo "-Wl,-Map=zephyr.map"; echo "-L ./include/generated"; echo "-u _OffsetAbsSyms -u _ConfigAbsSyms"; echo "-e __start"; echo "-Wl,--start-group"; echo "-Wl,--whole-archive"; echo ""; echo " src/built-in.o"; echo "libzephyr.a"; echo "-Wl,--no-whole-archive"; echo "./arch/arm/core/offsets/offsets.o"; echo "-Wl,--end-group"; echo "-L /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0 -L /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/.  -lgcc"; ) > zephyr.lnk
/home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-eabi/arm-poky-eabi-gcc -T linker.cmd @zephyr.lnk -o .tmp_zephyr.prebuilt
/home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/libexec/arm-poky-eabi/gcc/arm-poky-eabi/5.2.0/real-ld: error: .tmp_zephyr.prebuilt uses VFP register arguments, /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/libgcc.a(_udivmoddi4.o) does not
/home/marcus/work/zephyr/install/zephyr-sdk/sysroots/i686-pokysdk-linux/usr/libexec/arm-poky-eabi/gcc/arm-poky-eabi/5.2.0/real-ld: failed to merge target specific data of file /home/marcus/work/zephyr/install/zephyr-sdk/sysroots/armv5-poky-eabi/usr/lib/arm-poky-eabi/5.2.0/libgcc.a(_udivmoddi4.o)

[zephyrbot]

by Juro Bystricky:

Well, I think the problem is there are missing ARM libgcc.a libraries for each library variant.
I am looking into it.