Zephyr SDK ships with very outdated newlib for Xtensa

[pfalcon]

I'm working on elaborating POSIX subsystem in Zephyr and unifying its support across different config options (different libc's, different archs, etc.) One particular idea is to make sure that minlibs contained the same internal (implementation) headers as newlib, so that higher-level, public POSIX headers can work interchangeably with either libc. One example of such a header is sys/_timeval.h, which defines struct timeval in newlib. After introducing and switching to such header for minlibc and switching to it in (initial version of) zephyrproject-rtos/zephyr#15628 , I got a failure with qemu_xtensa and newlib: https://app.shippable.com/github/zephyrproject-rtos/zephyr/runs/40119/5/tests .

Turns out, newlib for xtensa in SDK 0.10.0 doens't have that header. But it was introduced in 2015: https://sourceware.org/git/gitweb.cgi?p=newlib-cygwin.git;a=commit;h=bb0159489785d577ad0b8061a1ba7956ee0f89d0 . That means that newlib as used by xtensa is very, very old.

Discrepancies like this make it very hard to go forward with POSIX subsystem development, as almost any change leads to failures from undermaintained architectures like xtensa.

[pfalcon]

@galak, I believe I heard about issue this issue with different newlib version, and here it hits again.

[pabigot]

Recording for posterity: Some research indicates that xtensa uses a custom newlib that was forked from newlib 2.0.0 around 2014-10-26 and has been unchanged since 2015-04-08. Updating to newlib 3.1.0 as used in the other targets seems unlikely.

I agree this is annoying as it requires special case code in Zephyr just to accommodate this platform, and has caused strife in reviews of zephyrproject-rtos/zephyr#16626. I suspect it will stay this way as long as xtensa is a supported platform.

[pfalcon]

@pabigot, as you did the research, perhaps you could elaborate on some points:

xtensa uses a custom newlib that was forked

Going from passive voice to active, who forked it? And where exactly? And what kind of "xtensa" is that which uses a custom newlib? (E.g., I hack xtensa in my own free time, and don't use any outdated forked newlib's.)

Updating to newlib 3.1.0 as used in the other targets seems unlikely.

For comparison, we have "minlibc" kind of libc in Zephyr, which doesn't have any custom xtensa version. Of course, that's a bit contrived example, as minlibc doesn't do anything which would require platform dependence. But some guys like @nashif, in addition (or "in replacement" is what they have in mind?) to 2 libc's we already have, want to bring 3rd: zephyrproject-rtos/zephyr#16451 (comment) . That also would not have any purposely outdated xtensa variant, or mixing up this 3rd one into the picture would be totally wrong IMHO.

seems unlikely.

Bottom line, that's fine. We just need to accept the fact Zephyr has very wide scope, and different subsystems are supported to a different extent for different targets. Here, the talk is that Xtensa target in its current shape is not ready for POSIX subsystem support. If we can't fix it now, we can remove it from CI for that subsystem. Again, "now". This can, and hopefully will, change later. But currently, need to workaround targets like Xtensa is detrimental to the further development of POSIX subsys (which needs a lot of work itself).

[pabigot]

Zephyr is using https://github.com/jcmvbkbc/newlib-xtensa.

I can't comment on alternative approaches or the potential of a standard newlib on xtensa.

[pfalcon]

Zephyr is using https://github.com/jcmvbkbc/newlib-xtensa.

Oh, then it should be easy to ask first-hand. Hey @jcmvbkbc, do you still work on Xtensa? Any comments on the concerns expressed above? (That your fork of newlib is rather outdated; is it really the case no/not enough things were submitted upstream?) Thanks.

[nashif]

see jcmvbkbc/newlib-xtensa#5

[pfalcon]

see jcmvbkbc/newlib-xtensa#5

So, the picture is now clear: Xtensa couldn't be supported in Zephyr beyond minlibc. However, an old fork was found, which kinda allowed to support newlibc somewhat. Well, more thorough use leads to issues. So, looking towards proposing to TSC to do the needful on a next case Xtensa poses an issue to POSIX subsys maintenance. (So far, we have 2 workarounds, which wasn't that bad, but one is still not merged, even though 2nd is exactly the same as 1st.)

[pabigot]

newlib 3.1.0 has some xtensa stuff, but the sdk build breaks because it's not a supported CPU.

[jcmvbkbc]

Any comments on the concerns expressed above? (That your fork of newlib is rather outdated; is it really the case no/not enough things were submitted upstream?)

That's true. I haven't touched it in a while. I guess I can update it to the most recent newlib and attempt to submit it upstream.

[nashif]

I guess I can update it to the most recent newlib and attempt to submit it upstream.

@jcmvbkbc that would be great, thanks. We can help with the testing.