Memory corruption for newlib-nano with float printf and disabled heap

[martinjaeger]

Describe the bug

We are facing a strange issue in our project based on STM32L072 with 20k of RAM. If certain features are enabled such that most of the RAM is consumed, float variables in printf statements (using newlib nano) get replaced by random junk characters. Printing of integers works fine. Also printk with recently added float support (cbprintf) works fine.

Example code:

printf("Junk: %.2f and maybe not: %d\n", 123.4F, 1234);

Results in:

Junk: <0xb0>@<break>
. <0x9d> and maybe not: 1234

To Reproduce

Haven't been able to generate a minimum working example to reproduce the issue, as it disappears if too much of the code is removed. However, it does not seem to be an issue in the application firmware itself. The issue happens in different threads and stack usage is still quite low (because I put all threads immediately into k_sleep(K_FOREVER) to exclude possible application firmware bugs):

I: Thread analyze:
I:  thread_analyzer     : unused 80 usage 432 / 512 (84 %)
I:  serial_thread_id    : unused 1160 usage 120 / 1280 (9 %)
I:  leds_thread         : unused 104 usage 152 / 256 (59 %)
I:  gsm                 : unused 1208 usage 216 / 1424 (15 %)
I:  control_thread_id   : unused 912 usage 112 / 1024 (10 %)
I:  idle 00             : unused 188 usage 68 / 256 (26 %)
I:  main                : unused 792 usage 232 / 1024 (22 %)

Possible root cause and workaround

Our application doesn't use the heap. Since PR #28486, the RAM reserved for the heap seems to be garbage-collected away in that case (independent of the value of CONFIG_HEAP_MEM_POOL_SIZE) and can be reused for the stack.

However, newlib requires malloc if printf is used with %f: http://www.nadler.com/embedded/newlibAndFreeRTOS.html

If I add a line void *mem_test = k_malloc(4); to the code, Zephyr compiles in the heap management again and the issue is gone.

I'm not 100% sure if the above is really the root cause or if it made the issue disappear by coincidence, but it looks plausible to me. Maybe someone with more insight into newlib internals can confirm.

This link posted by @pabigot on Slack might also be relevant: https://stackoverflow.com/questions/28746062/snprintf-prints-garbage-floats-with-newlib-nano

Ping @nashif @dcpleung @andrewboie @carlescufi as you were involved in mentioned PR.

[andyross]

IIRC the newlib heap and the Zephyr system heap aren't the same memory, right? Maybe the bug here is a linkage problem where the newlib heap memory is clobbering something adjacent?

[martinjaeger]

Yes, that could be the case. Do you know how I can find out which memory location newlib uses as its heap?

[andyross]

I dug around here a little, but obviously without an exerciser there's not a lot I can do. In fact yes, as I (vaguely) remember the newlib malloc() heap is an sbrk-style thing that is a physically separate region of memory from the Zephyr heap. So despite the fact that they both say "malloc" these regions shouldn't interact at all.

Can you post the zephyr.map files from your minimally-changed "working" and "failing" cases? I still strongly suspect this is going to turn out to be some kind of linker interaction, plausibly alignment of the heap that interacts with pointer math somewhere in the soft float implementation?

[martinjaeger]

@andyross sorry for the delayed response. Since my upgrade to Zephyr SDK v0.12.0 I was not able to reproduce the issue anymore. Instead, I get the following error printed to the serial console now:

assertion "Balloc succeeded" failed: file "/workdir/build/build_arm/.build/arm-zephyr-eabi/src/newlib/newlib/libc/stdlib/mprec.c", line 778
exit

A colleage compiled our firmware with exactly the same settings on Windows and could still reproduce the issue. So here is a bunch of map files: https://nextcloud.libre.solar/s/CjeXZZt7AYKjEBJ

• zephyr_balloc_assert.map: Compiled with SDK v0.12 and leading to above error
• zephyr_junk_char.map: Same firmware configuration compiled under Windows, leading to junk character output
• zephyr_no_junk.map: Configuration slightly changed such that the floats are printed correctly (also compiled under Windows)

The issue seems to be independent of the Zephyr version. I get the same behavior with v2.4-branch and v2.5-RC1.

[martinjaeger]

Some additional information: The newlib version of my colleague where the issue appeared is 3.1.0, where as Zephyr SDK v0.12 updated newlib to 3.3.0.

This seems to be relevant to the problem: https://census-labs.com/news/2020/01/31/multiple-null-pointer-dereference-vulnerabilities-in-newlib/

So maybe the remaining problem is only to create a linker error if not enough space is left for the newlib heap, if that makes sense?

[andyross]

Grooming bugs: sorry, can you confirm that a 3.3.0 newlib fixes the problem? Or you just suspect it might?

[andyross]

And serendipitously, I just ran into #33164 which very well might be the root cause if your application has multiple threads trying to use that heap.

[martinjaeger]

Grooming bugs: sorry, can you confirm that a 3.3.0 newlib fixes the problem? Or you just suspect it might?

I would not say it fixes the problem completely. I can confirm that we don't see the junk characters anymore with newlib 3.3.0. But we get above assertion message instead. Ideally, the linker would claim sufficient memory for newlib such that we see at compile-time that we run out of space.

Regarding the potential race condition in newlib: I'm not sure if that's really the root cause. I suspended all threads except for one with k_sleep(K_FOREVER). So there should not be multiple threads calling printf at the same time. I'm happy to verify it once a fix for newlib synchronization is in place.

[github-actions]

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.

[galak]

@martinjaeger Can you re-test this to see if #35227 might have helped.