Newlib: inconsistent usage of write() vs _write() in builds for different architectures to dump error messages to console

[pfalcon]

Newlib build included with x86 toolchain calls write() in [TBF], while e.g. arm - _write(). This is inconsistent and causes confusion. x86 way of calling write() also requires it to be defined, but default implementation (without POSIX subsystem enabled) is able to write only to a console. But people don't know that, and call it e.g. on sockets, with output dumped to console. All these problems could be avoided in x86 build also used _write() (the whole idea of underscore-prefixed function is that "it works kind like posix, but doesn't support full posix functionality").

[pfalcon]

This is submitted as offshot of zephyrproject-rtos/zephyr#25443 (comment) . cc @cfriedt .

More information is expected to follow.

[galak]

Can you clarify, when you say the toolchain calls write(), what do you mean? Trying to understand what the function call chain you are seeing on x86 that differs from arm.

[pfalcon]

Not toolchain, but pre-built Newlib static lib as included with the Zephyr SDK toolchains. I'll post steps to reproduce a bit later.

[galak]

Sure, I guess having a backtrace comparison between x86 and arm would be useful to see the difference.

[pfalcon]

Well, I'm not sure about backtrace, the best way to expose an issue is link-time error. So, let's comment definitions of both write() and _write() in https://github.com/zephyrproject-rtos/zephyr/blob/master/lib/libc/newlib/libc-hooks.c#L183 . The diff is in top commit of https://github.com/pfalcon/zephyr/commits/newlib-write-vs-_write (pfalcon/zephyr@65dafeb as of now, may be rebased).

Then let's build samples/net/sockets/big_http_download (choice is arbitrary, it just well exposed issue, based on initial CI failures in zephyrproject-rtos/zephyr#25443).

$ west build -b qemu_x86
...
/mnt/hdd/opt/zephyr-sdk-0.11.2/x86_64-zephyr-elf/bin/../lib/gcc/x86_64-zephyr-elf/9.2.0/../../../../x86_64-zephyr-elf/bin/ld: zephyr/lib/libc/newlib/liblib__libc__newlib.a(libc-hooks.c.obj): in function `_exit':
/home/pfalcon/projects-3rdparty/Embedded/Zephyr/zephyr/lib/libc/newlib/libc-hooks.c:242: undefined reference to `_write'
/mnt/hdd/opt/zephyr-sdk-0.11.2/x86_64-zephyr-elf/bin/../lib/gcc/x86_64-zephyr-elf/9.2.0/../../../../x86_64-zephyr-elf/bin/ld: /home/pfalcon/opt/zephyr-sdk-0.11.2/x86_64-zephyr-elf/x86_64-zephyr-elf/lib/32/libc.a(lib_a-chk_fail.o): in function `__chk_fail':
chk_fail.c:(.text.__chk_fail+0x73): undefined reference to `write'
/mnt/hdd/opt/zephyr-sdk-0.11.2/x86_64-zephyr-elf/bin/../lib/gcc/x86_64-zephyr-elf/9.2.0/../../../../x86_64-zephyr-elf/bin/ld: /home/pfalcon/opt/zephyr-sdk-0.11.2/x86_64-zephyr-elf/x86_64-zephyr-elf/lib/32/libc.a(lib_a-writer.o): in function `_write_r':
writer.c:(.text._write_r+0x1d): undefined reference to `write'

So, what we see here is that write() (without underscore) gets called (among Zephyr in-tree cases) from:

• zephyr-sdk-0.11.2/x86_64-zephyr-elf/x86_64-zephyr-elf/lib/32/libc.a(lib_a-chk_fail.o): in function `__chk_fail'
• zephyr-sdk-0.11.2/x86_64-zephyr-elf/x86_64-zephyr-elf/lib/32/libc.a(lib_a-writer.o): in function `_write_r'

Which is Newlib prebuilt libc.a as shipped with Zephyr SDK (0.11.2, as can be see).

Let's compare with errors for an Arm board:

$ west build -b frdm_k64f
...
/mnt/hdd/opt/zephyr-sdk-0.11.2/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/9.2.0/../../../../arm-zephyr-eabi/bin/ld: zephyr/lib/libc/newlib/liblib__libc__newlib.a(libc-hooks.c.obj): in function `_exit':
/home/pfalcon/projects-3rdparty/Embedded/Zephyr/zephyr/lib/libc/newlib/libc-hooks.c:242: undefined reference to `_write'
/mnt/hdd/opt/zephyr-sdk-0.11.2/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/9.2.0/../../../../arm-zephyr-eabi/bin/ld: /home/pfalcon/opt/zephyr-sdk-0.11.2/arm-zephyr-eabi/arm-zephyr-eabi/lib/thumb/v7e-m/nofp/libc_nano.a(lib_a-writer.o): in function `_write_r':
writer.c:(.text._write_r+0x10): undefined reference to `_write'

So, we now see that prebuilt Newlib tries to call _write() (with underscore), specifically from:

• zephyr-sdk-0.11.2/arm-zephyr-eabi/arm-zephyr-eabi/lib/thumb/v7e-m/nofp/libc_nano.a(lib_a-writer.o): in function `_write_r'

That pinpoints the issue reported: when built for different architectures, the same module within Newlib (specifically, lib_a-writer.o: _write_r) calls either write or _write, depending on the architecture. Again, this leads to the necessity to define both on Zephyr side, and defining write leads to the user confusion (for the case of CONFIG_POSIX_API=n, which is the default, and the focus of our discussion here).

But looking closer, we see that the situation is even more tangled: for different architectures (or straight, boards), different Newlib variants are used, e.g. for qemu_x86 "full" Newlib is used by default, while for frdm_k64f - "nano".

It's easy to imagine that such a difference in Newlib variants was done as a "feature" and with the best intention, but it ends up just another inconsistency among platforms/boards.

[pfalcon]

Ok, so given that we spotted "full" vs "nano" Newlib discrepancy, let's check whether write vs _write discrepancy is actually caused by it:

--- a/samples/net/sockets/big_http_download/prj.conf
+++ b/samples/net/sockets/big_http_download/prj.conf
@@ -1,5 +1,6 @@
 # General config
 CONFIG_NEWLIB_LIBC=y
+CONFIG_NEWLIB_LIBC_NANO=n

/mnt/hdd/opt/zephyr-sdk-0.11.2/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/9.2.0/../../../../arm-zephyr-eabi/bin/ld: /home/pfalcon/opt/zephyr-sdk-0.11.2/arm-zephyr-eabi/arm-zephyr-eabi/lib/thumb/v7e-m/nofp/libc.a(lib_a-writer.o): in function `_write_r':
writer.c:(.text._write_r+0x1a): undefined reference to `_write'

So, nope, the original hypothesis appear to hold - the discrepancy is per-architecture, not per-Newlib-variant.

[pfalcon]

To elaborate on the choices we have, until we have this fixed, we have to apply pretty ugly workarounds like zephyrproject-rtos/zephyr#25479 (where we effectively say "we can't get it right, so d'oh, it's your problem"). Another alternative is to keep good people like @cfriedt confused (which is the current situation).

[cfriedt]

From the prebuilt toolchain (version 0.11.2):

~/zephyr-sdk-0.11.2/aarch64-zephyr-elf$ nm $(find . -name '*.a') | grep "\( write\| _write\| _write_r\)$" | grep -v
 "_ZN\|_ZZ\|_ZS\|_ZT\|_ZG\|_IT" | grep -v "^  " | grep -v ":" | sort -u
0000000000000000 T write
0000000000000000 T _write
0000000000000000 T _write_r
00000000 T write
00000000 T _write
00000000 T _write_r

~/zephyr-sdk-0.11.2/arc-zephyr-elf$ nm $(find . -name '*.a') | grep "\( write\| _write\| _write_r\)$" | grep -v "_ZN\|_ZZ\|_ZS\|_ZT\|_ZG\|_IT" | grep -v "^  " | grep -v ":" | sort -u
00000000 T write
00000000 T _write
00000000 T _write_r

~/zephyr-sdk-0.11.2/arm-zephyr-elf$  nm $(find . -name '*.a') | grep "\( write\| _write\| _write_r\)$" | grep -v "_ZN\|_ZZ\|_ZS\|_ZT\|_ZG\|_IT" | grep -v "^  " | grep -v ":" | sort -u
00000000 T write
00000000 T _write
00000000 T _write_r
00000000 W _write
00000001 T write
00000001 T _write
00000001 T _write_r
00000001 W _write
000000f5 T _write
000001e8 T _write

~/zephyr-sdk-0.11.2/nios2-zephyr-elf$ nm $(find . -name '*.a') | grep "\( write\| _write\| _write_r\)$" | grep -v "_ZN\|_ZZ\|_ZS\|_ZT\|_ZG\|_IT" | grep -v "^  " | grep -v ":" | sort -u
00000000 T write
00000000 T _write_r

~/zephyr-sdk-0.11.2/riscv64-zephyr-elf$ nm $(find . -name '*.a') | grep "\( write\| _write\| _write_r\)$" | grep -v "_ZN\|_ZZ\|_ZS\|_ZT\|_ZG\|_IT" | grep -v "^  " | grep -v ":" | sort -u
0000000000000000 T write
0000000000000000 T _write
0000000000000000 T _write_r
00000000 T write
00000000 T _write
00000000 T _write_r

~/zephyr-sdk-0.11.2/sparc-zephyr-elf$ nm $(find . -name '*.a') | grep "\( write\| _write\| _write_r\)$" | grep -v "_ZN\|_ZZ\|_ZS\|_ZT\|_ZG\|_IT" | grep -v "^  " | grep -v ":" | sort -u
00000000 T write
00000000 T _write_r

~/zephyr-sdk-0.11.2/x86_64-zephyr-elf$ nm $(find . -name '*.a') | grep "\( write\| _write\| _write_r\)$" | grep -v "_ZN\|_ZZ\|_ZS\|_ZT\|_ZG\|_IT" | grep -v "^  " | grep -v ":" | sort -u
0000000000000000 T write
0000000000000000 T _write_r
00000000 T write
00000000 T _write_r

~/zephyr-sdk-0.11.2/xtensa$ nm $(find . -name '*.a') | grep "\( write\| _write\| _write_r\)$" | grep -v "_ZN\|_ZZ\|_ZS\|_ZT\|_ZG\|_IT" | grep -v "^  " | grep -v ":" | sort -u
00000000 T write

[pfalcon]

@cfriedt: Yep. Maybe if you're on it, you could grep libs for other archs.

[pfalcon]

Maybe if you're on it, you could grep libs for other archs.

@cfriedt, thanks! Mind that when you post a new comment, people involved a ticket get notification (and that's how most follow multitude of their tickets, I guess), but when you edit, they don't. It's certainly not a problem here, thanks for finding time to do that. Just e.g. in PR reviews, it makes sense to post new info as a new comment (or if it makes more sense to edit existing, to post a short notice that previous info was udpated). Thanks.

Anyway, looking at the details above, it's clear that x86 is not the only arch which calls write(). As far as I can tell, it's roughly 50/50 among archs.

Ok, when other tasks permit, I'll have a look at Newlib source to figure out if something can be done about it.