newlib: enable long long support

[galak]

Signed-off-by: Kumar Gala [email protected]

[pfalcon]

Ok, so I tested printf("Hello World!\n"); with SDK 0.10.2 vs sdk-0.10.3-pr-101:

qemu_cortex_m3:

   text	   data	    bss	    dec	    hex	filename
  13594	   1436	   3940	  18970	   4a1a	zephyr.elf

   text	   data	    bss	    dec	    hex	filename
  27234	   1800	   3940	  32974	   80ce	zephyr.elf

(Didn't test qemu_x86 which has MMU and other bloat enabled, making results much less conclusive.)

So yeah, it's huge code increase. And as much as dislike bloat, I'd give this patch +0.75. The whole situation is why we have minlibc. Let it continue to be, and let it be the default. And let's continue to carefully craft and optimize it. But we need the other opposite of spectrum - a libc which just allows existing real-world apps to work on Zephyr, without being swamped with patching them.

[pfalcon]

I'd give this patch +0.75

Well, I don't know people who'd be strict on unbloated code size and would counter my arguments above. Let me randomly ping @nashif and @mike-scott to spread awareness.

[pfalcon]

Well, I don't know people who'd be strict on unbloated code size and would counter my arguments above.

Posted RFC to the mailing list: https://lists.zephyrproject.org/g/devel/message/6201

[pabigot]

I wanted to see what the current state is across toolchains. Zephyr's SDK is built only with newlib nano, unlike Arm Embedded which builds both standard and nano newlib and allows selection at build time. I ran the sample below under various configurations using the particle_xenon (ARM NRF52840).

ZTC is Zephyr SDK 0.10.3
ZPR is Zephyr SDK 0.10.3-PR-101
AETC is Arm Embedded 8-2019-q3

Flash and SRAM space and increases relative to 0.10.3 minimal libc:

TC	libc	FLASH	#^	%^	SRAM	#^	%^
ZTC	minimal	16204	0	0	3944	0	0
ZTC	newlib	20076	3872	23.9	4636	692	17.5
ZPR	minimal	16204	0	0	3944	0	0
ZPR	newlib	33380	17176	106	5000	1056	26.8
AETC	minimal	16256	52	0.32	3944	0	0
AETC	nano	18764	2560	15.8	4096	152	3.85
AETC	newlib	44648	28444	176	6532	2588	65.6

Relative to 0.10.3 PR101 adds 13304 bytes FLASH and 364 bytes SRAM in a newlib configuration.

Also it's clear that PR101 does not enable all the newlib features of Arm Embedded, since that adds another 11 K FLASH and 2.5 K RAM.

I don't know what newlib nano has that minimal libc doesn't, but adding an option for newlib nano to the Zephyr SDK might catch a few cases where minimal isn't good enough, but full is too big. Newlib also has a policy for what's in nano, avoiding arguments about any additions.

Test Program

#include <zephyr.h>
#include <stdio.h>
#include <inttypes.h>

void main(void)
{
	u64_t value = (0x12345678LL << 12);
	printf("Hello World! %s %16" PRIx64 "\n", CONFIG_BOARD, value);
}

Zephyr toolchain 0.10.3, minimal libc:

Memory region         Used Size  Region Size  %age Used
           FLASH:       16204 B         1 MB      1.55%
            SRAM:        3944 B       256 KB      1.50%
        IDT_LIST:          56 B         2 KB      2.73%

Output:

***** Booting Zephyr OS build v2.0.0-rc1-204-gf8b068ed88f2 *****
Hello World! particle_xenon %lx

Zephyr toolchain 0.10.3, newlib libc:

Memory region         Used Size  Region Size  %age Used
           FLASH:       20076 B         1 MB      1.91%
            SRAM:        4636 B       256 KB      1.77%
        IDT_LIST:          56 B         2 KB      2.73%

Output:

***** Booting Zephyr OS build v2.0.0-rc1-204-gf8b068ed88f2 *****
Hello World! particle_xenon                lx

Zephyr toolchain 0.10.3-PR-101, minimal libc:

Memory region         Used Size  Region Size  %age Used
           FLASH:       16204 B         1 MB      1.55%
            SRAM:        3944 B       256 KB      1.50%
        IDT_LIST:          56 B         2 KB      2.73%

Output:

**** Booting Zephyr OS build v2.0.0-rc1-204-gf8b068ed88f2 *****
Hello World! particle_xenon %lx

Zephyr toolchain 0.10.3-PR-101, newlib libc:

Memory region         Used Size  Region Size  %age Used
           FLASH:       33380 B         1 MB      3.18%
            SRAM:        5000 B       256 KB      1.91%
        IDT_LIST:          56 B         2 KB      2.73%

Output:

***** Booting Zephyr OS build v2.0.0-rc1-204-gf8b068ed88f2 *****                                                     
Hello World! particle_xenon      12345678000

Arm Embedded 8-2019-q3-update, minimal libc

Memory region         Used Size  Region Size  %age Used
           FLASH:       16256 B         1 MB      1.55%
            SRAM:        3944 B       256 KB      1.50%
        IDT_LIST:          56 B         2 KB      2.73%

Output:

***** Booting Zephyr OS build v2.0.0-rc1-204-gf8b068ed88f2 *****
Hello World! particle_xenon %lx

Arm Embedded 8-2019-q3-update, newlib nano libc

Memory region         Used Size  Region Size  %age Used
           FLASH:       18764 B         1 MB      1.79%
            SRAM:        4096 B       256 KB      1.56%
        IDT_LIST:          56 B         2 KB      2.73%

Output:

***** Booting Zephyr OS build v2.0.0-rc1-204-gf8b068ed88f2 *****
Hello World! particle_xenon                lx

Arm Embedded 8-2019-q3-update, newlib libc

Memory region         Used Size  Region Size  %age Used
           FLASH:       44648 B         1 MB      4.26%
            SRAM:        6532 B       256 KB      2.49%
        IDT_LIST:          56 B         2 KB      2.73%

Output:

***** Booting Zephyr OS build v2.0.0-rc1-204-gf8b068ed88f2 *****
Hello World! particle_xenon      12345678000

[carlescufi]

@pabigot thanks for the analysis.
Regarding newlib nano, I thought @galak said that it was deprecated and the changes were folded back to newlib with compile options?

[pabigot]

I don't see anything in upstream newlib about nano being deprecated. Being able to select a libc with no more features than the application needs is a real advantage.

The nice thing about nano is that it's supported in the GNU toolchain as a build-time option by overriding the GCC driver behavior with a single argument. We could, in theory, support any number of custom libc variants with different features using the same technique, but that seems out of scope for a general purpose toolchain.

[galak]

I don't see anything in upstream newlib about nano being deprecated. Being able to select a libc with no more features than the application needs is a real advantage.

The nice thing about nano is that it's supported in the GNU toolchain as a build-time option by overriding the GCC driver behavior with a single argument. We could, in theory, support any number of custom libc variants with different features using the same technique, but that seems out of scope for a general purpose toolchain.

Not sure where the thought that nano was deprecated.

I was thinking it might be worth looking at adding support for long long to nano-libc. Not sure what else we get from the "fuller" newlib. Change the build to support both nano-libc & "fuller" is work that would need to be done to crosstool-ng.

[carlescufi]

OK I think the confusion from my side stems from the fact that perhaps newlib nano used to be a separate project and it is instead now part of GCC?

[pabigot]

Yes, newlib-nano was originally a fork of newlib (separate from GCC), but its features were merged upstream as configure options back in 2014.

[mike-scott]

@pfalcon If a user falls in the "I want code compatibility" use-case, I can see the addition of long long support being very attractive. However, if I was just trying to use tinycbor (which "selects" NEWLIB_LIBC) and I suddenly found out my sample jumped 13k in size, I would be a bit disgruntled.

@pabigot This is a great analysis. I agree that an option to build newlib nano might need to be explored.

[pfalcon]

@pabigot: Just as as everyone, I thank you for detailed analysis. And just as everyone, I think that we should have more, many more configuration options to satisfy everyone's wishes. But that's quite a separate matter, with its own requirements, analysis phase (much more detailed than provided here), implementation effort, then review effort.

Here we however have much simpler task at hand - decide whether to apply this patch now, or not. (For reference, this feature implementation cycle is already 4 months, if counting from #62 creation.)

[pfalcon]

If a user falls in the "I want code compatibility" use-case, I can see the addition of long long support being very attractive. However, if I was just trying to use tinycbor (which "selects" NEWLIB_LIBC) and I suddenly found out my sample jumped 13k in size, I would be a bit disgruntled.

@mike-scott, I fully agree that it's sad to to get 13K bump in size. But thinking more about it, the root of sadness is that "tiny" libraries depend on the whole bloated C library. But let's not indulge in sadness, and look at the bright side of things. My interpretation of why tinycbor depends on Newlib is that "people want to write software which uses good deal of libc functionality". But then it's easy to imagine to that there're even more software which wants to use even more of libc.

So, we just need to make a decision about this patch - whether we want to set polar bounds of libc support in Zephyr (full Newlib vs minlibc), and let interested parties to fill choices inbetween (with a usual warning about premature optimization), or stay at the adhoc point we're now.

[nashif]

Need to look into supporting multiple libc configurations, this patch is going to break many users and applications that are on the limits of flash and ram and is the benefit is minimal (we did not need that the last 5 years, why change now?).

[pfalcon]

(we did not need that the last 5 years, why change now?).

Because nobody used Zephyr, and people only start to use it now, and discover a lot of issues.

[pfalcon]

Note that one of the usages for %lld formatting specifier is to print unix time, e.g. for various networking protocols. It's well known by now that this time cannot be represented as 32-bit value, as that leads to Year 2038 problem. So, unix time must be at least 64-bit value, and its printing consequently should be done using %lld formatting specifier.

So, this is (or borders on) a security issue, so adding @d3zd3z into loop.

(Yeah, it's me just thinking what to do if this patch doesn't go thru in regard to GoogleIoT SDK porting. Like, me coming to Google folks and saying "Hey, you recently fixed a security issue in your codebase. Would-be CVE, or maybe actual CVE. But would you be so kind to add an option to revert back to the original insecure behavior, just for the sake of Zephyr RTOS?" Would be a fun conversation.)

[mike-scott]

@pfalcon CBOR data formatting is now included in the v1.1 LwM2M spec. I'm looking to remove the newlib dependency of tinycbor so that the smaller boards will be able to move forward. I appreciate you're newly found optimism towards better portability in Zephyr, but I'm still not convinced the average board maintainer will be "OK" with this change as it is.

That being said, I would be equally frustrated if I was working on the GoogleIoT stuff. :/. Unfortunately, I see this PR as being linked to a "small" newlib vs. "larger" newlib config option.

[pfalcon]

but I'm still not convinced the average board maintainer will be "OK" with this change as it is.

Then we need specific review votes (from "board maintainers").

[pfalcon]

Confirming that this patch solved the issue was GoogleIoTSDK - where previously it just crashed, now it works as expected.

[bogdanm]

Adding this as a critical issue for me too. I'm trying to build a firmware based on Lua 5.3 and Zephyr OS, which breaks in many wonderful ways due to the lack of long long support in Newlib. Using the minimal libc is quite difficult with Lua. It's quite funny really: floating point support looks fine, but no long longs.

EDIT: I found an workaround by using ZEPHYR_TOOLCHAIN_VARIANT=gnuarmemb and a GNU Arm Embedded toolchain installed from developer.arm.com (which handles long longs fine)

[stephanosio]

After #152 gets merged, we should consider enabling CT_LIBC_NEWLIB_IO_LL (this will add long long support only to the normal variant without touching the nano variant).

N.B. with the changes in #152, we can build normal (lib{c,g}.a) and nano (lib{c,g}_nano.a) variants of newlib separately.

[stephanosio]

Force pushing new commit, as #153 is now merged:

This commit enables 'long long' (64-bit) support in the newlib.

Note that 'long long' support is only enabled for the normal newlib
(libc.a), without affecting the newlib nano variant (libc_nano.a).

Signed-off-by: Stephanos Ioannidis <[email protected]>

[stephanosio]

Running hello_world with printf on qemu_cortex_m3:

newlib normal (CONFIG_NEWLIB_LIBC=y and CONFIG_NEWLIB_LIBC_NANO=n)

   text    data     bss     dec     hex filename
  41958    2836    3988   48782    be8e zephyr/zephyr.elf

newlib nano (CONFIG_NEWLIB_LIBC=y and CONFIG_NEWLIB_LIBC_NANO=y)

   text    data     bss     dec     hex filename
  12394     880    3940   17214    433e zephyr/zephyr.elf