Console, logger, shell architecure

[nordic-krch]

Relations between consoles, logger and shell must be clarified as it generates a lot of confusion as it is now.

Topics to clarify:

• what is console?
• Is it a 'standard io' and thus should be single instance or it's just an 'io' but there is one 'default' (the console).
• What's the purpose of console? Any other use cases than logging and shell?

Current issues with console:

• byte API - slow. As logger backend candidate throughput must be significantly improved.
• multiple consoles can be enabled but actually one will be used (__stdout_hook_install)
• as console is a relatively low level module it is strange to see references to mcumngr in uart_console.c

It is somehow related to #10480. User expects that enabling shell commands in a module will lead to shell being enabled but as shell is multi-instance it should enable shell with console backend.

[nordic-krch]

@pfalcon @nashif @carlescufi @jarz-nordic @pizi-nordic @jukkar @aescolar could you comment?
I feel like we should agree on that as way forward is not settled.
@pfalcon you are working around console, maybe you already have this things somehow figured, could you share?

[aescolar]

One comment here: there is 2 quite different "consoles" in Zephyr.
drivers/console which was a set of backends, used by printk and the old shell.
and subsys/console (CONSOLE_SUBSYS) which was not used inside zephyr itself, but AFAIK pfalcon was promoting as the API to have.

[pfalcon]

One comment here: there is 2 quite different "consoles" in Zephyr.

That's right. Here's a short intro into the situation:

The name of "drivers/console" would suggest that those should be drivers, and of course the client applications should not call into drivers. Instead, apps would go thru "subsystem". That's the pattern we follow in the rest of Zephyr, even if many subsystems are as simple as multiplexors from struct device to virtual methods define by a driver for a device.

But drivers/console were special - the apps actually called into particular drivers, and to add support of new driver, one had to patch an application!

That's not the biggest problem with drivers/console though - the API there is rather peculiar. It's line-based, and line editing is handled by the driver! Yes, UART driver handles all those left/right arrow key escape sequences, somewhere straight in the ISR. Needless to say, that line-editing doesn't work for other "console" "drivers" (unless they duplicate code). Another peculiarity was that API was fifo-based, where an app must manage and setup all the buffers for input lines.

There were some other issues, like problems with interrupt handling/buffering, causing character loss on paste from host.

[pfalcon]

So, to port existing applications to Zephyr, I had a requirement to provide ANSI C/POSIX style functions:

• gets()-alike, to call it and get a line from console
• getchar()-like, to call and get a single char

That's how subsys/console was started. Yes, in a sense, it's "userspace" API, because Zephyr internal components were (and remain) too "originally-designed" to use simple APIs like those.

To implement gets-alike (sane one, called getline()), I was able to base on the API of existing drivers. Implementation of getchar() required working around implementations of the (UART) driver, which soon after was broken. I had little choice as to stop using drivers/console for getchar() at all, and write own "driver" for UART, which so far resides in subsys/console.

[pfalcon]

So, that's the situation we have:

• Drivers with adhoc, inflexible API. But we have a bunch of them, e.g. "console" "driver" for telnet!
• A higher-level subsystem with well-known, user-friendly API, which tries to workaround existing console drivers as hard as it can, with consequences. E.g., an app may choose to use either getline() or getchar(), getline() would go thru existing drivers, but getchar() is backed only by UART (but works with it well - well, for me, there're some reports from other people).

[aescolar]

My 2 cents:

• It would have been nice if the shell & logger would have used the console, as they would have inherited the already available backends. That would have saved work for others, but I can understand they did not fit. From my point of view, no point discussing this.

• About the shell: AFAIK, it seems to assume you always want the embedded Zephyr device to do the shell line editing, and all the fancy stuff which comes with it. If you have a raw, compatible terminal to interact with, and enough RAM and cycles in the embedded device, that works. But, it would seem pretty problematic in other cases. Say, where due to transport latency you want to let the remote terminal handle the line editing, and you only want to transport full lines to the shell. Or say, when you cannot just move the cursor back in the output device.
  Meaning, the impression I got is that the shell design is very tailored to a VT100 compatible terminal over UART.

I, personally, do not mind how the shell/logger are architected while they do easily support different backends.
But when it was not working for native_posix, I just got the impression that for the shell, it was way easier to emulate a UART plugged to a raw terminal than to try to get another backend for it working.

[jakub-uC]

@aescolar : Thank you for this input. I will work to improve shell according to your suggestions but it will take time... My priority is to fix bug reported by Anas: #10413

[pfalcon]

So, to not push deeds behind the talk (because indeed, we're talking long enough on that, it started long before this ticket), #10765 was the next step.

Capturing @nordic-krch's comments to it:

as console will be logger/shell backend it should be possible to reconfigure it to synchronous mode in case of panic when data is flushed. In that mode console must not use interrupts or kernel primitives. I only see use case for writing data so it may be that write_sync() would be sufficient. Another option would be to add dedicated function for re-enabling in blocking mode.

Ack from my side on the requirement, we can think what's the cleanest (re: POSIX integration too) and the most general way to implement it.

[nordic-krch]

My comment: #10765 (comment) is also relevant to that discussion. It's about handling receiver errors on api like the one in #10765:

How do you see reporting, of RX buffer overflow? It is not hw error but rather application not reading data on time. In that case tty has valid data buffered but tty_read returns error?

[pkral78]

Some notes:
#10247 was good start, but we should also add support for polled (non-isr) uart drivers. For example RTT console driver is one that must be implemented by that way. And in fact #10709 implemented it already for shell subsystem - which is unnecessary effort as my POV is, that shell subsystem could uses console subsystem by default.

Also it would be useful if console subsystem supports multiple ttys on multiple uarts (outside default on for stdin/stdout).