Zephyr Input Subsystem proposal

[fabiobaltieri]

NOTE: the details are already out of date, core PR: #54908

TL:DR;

Linux-like events for Zephyr, proof of concept here: #54620

Introduction

Implement a subsystem in upstream Zephyr that can handle input events from various types of input devices and distribute them to other threads in the application.

Problem description

Zephyr is lacking a generic input subsystem. Right now the closest thing is the kscan api, which is only meant to support keyboard matrixes and has been somewhat abused to implement touchscreens and other input devices.

Meanwhile devices that didn't quite fit that model somehow found their way into other subsystems, for example the "nrfx_qdec" driver, commonly used for scroller encoders ended up as a sensor rotation device, meaning that the "clicks" gets an angle conversion, just to convert the angle back to clicks in the application.

Lastly, input devices that would not fit any subsystem started to declare their own API as if they were a subsystem on their own, with more proposals for device specific APIs or very specific functionalities.

Requirements

• Support a many-to-many model, that is multiple input devices can send events to many different listeners, use iterable sections for registering the listeners for modularity
• Be flexible enough to support any input device (buttons, keyboards, mice, touchscreens etc...) without bloating the API (no specialized functions)
• Work as a replacement for the current kscan API, provide a backward compatibility layer, be usable in the current drivers, with current applications
• Provide a way to extend the basic functionalities (long press, double click...) in a generic way

Proposed change

Implement a Zephyr input subsystem inspired by the Linux input events model.

Linux supports input devices using an input event subsystem. Each device registers itself with the system (causing the "/dev/inputX" node to be created among other things), sets the supported capabilities, and then can start sending events whenever available.

An event describes a single entity within the input device, may that be a button, a relative or absolute movement over a single axis or even a scancode).

Each burst of events is terminated by a synchronization marker, which is used to indicate that the last set of events is in a stable state and can be used. This allows representing for example an X-Y coordinate and multi-tracking for a touchscreen device.

Detailed RFC

The proposed Zephyr implementation is conceptually similar to the Linux one, but somewhat simplified with the idea that a Zephyr application would be more resource constrained, but also be exposed to a lower number of devices and listeners.

There's no special device registration and no capabilities, but the device identifier is passed down with the event, which would looks like this:

struct input_event {
        const struct device *dev;
        uint16_t type; /* key, relative, absolute... */
        uint16_t code; /* button ID, axis name (x/y/z)... */
        int32_t value; /* 0/1 for keys, coordinate for axis... */
};

Events are either handled by a single message queue and a dedicated "input" thread, or no queue at all, in which case listeners are called synchronously. This allows simple use cases to just get all input events in the system, or complex use cases to register a synchronous notifier and then pass that to their own event system.

Listeners registers a callback in an iterable section and can optionally request to get notified for events of only a specific device:

static void input_cb(struct input_event *evt, bool sync)
{
        /* get the events for all devices */
}
INPUT_LISTENER_CB_DEFINE(NULL, input_cb);

static void buttons_cb(struct input_event *evt, bool sync)
{
        /* just get events for the buttons device */
}
INPUT_LISTENER_CB_DEFINE(DEVICE_DT_GET(DT_NODELABEL(buttons)), buttons_cb);

Device drivers can send events in the queue at any time, the format and identifiers are similar to the Linux ones

if (pressed) {
        input_report_abs(dev, INPUT_ABS_X, row, false, K_FOREVER);
        input_report_abs(dev, INPUT_ABS_Y, col, false, K_FOREVER);
        input_report_key(dev, INPUT_BTN_TOUCH, 1, true, K_FOREVER);
} else if (pressed_old && !pressed) {
        input_report_key(dev, INPUT_BTN_TOUCH, 0, true, K_FOREVER);
}

The timeout argument is passed straight to "k_msgq_put" if that's used, ignored otherwise. TBD what to do with it, maybe make it a subsystem or per-device option instead.

Synchronization is done using the MSb of "evt->type" to avoid using an entire event message just for the sync bit.
The type and code are also copied from Linux, tentatively using the same codes as well.

Use cases

A common use case for input devices is to have multiple drivers generating events (buttons, encoders...) and then have them consumed by multiple listeners (USB, Bluetooth, some internal logic for handling power management, pairing etc).

For the EC use case this would provide a simple clean API to aggregate events from multiple devices (GPIO buttons, lid switches, keyboard matrix...) and distribute them to other tasks for various functionalities (8042, MKBP, backlight/brightness hw control...), replacing the "keyboard_state_changed" and "mkbp_keyboard_add" functions for passing the state between keyboard driver and AP channel.

Extensions and compatibility

The API should be flexible enough to be used to support further input event processing for common use cases. This can be done by defining a devicetree entry for the extension which specifies the configuration parameters and the input device. For example, one could define a filter to further process key events to support long presses:

buttons: buttons {
        compatible = "zephyr,gpio-keys";
        button0: button_0 {
                gpios = <&gpio0 13 (GPIO_PULL_UP | GPIO_ACTIVE_LOW)>;
                label = "Push button switch 0";
                zephyr,code = <INPUT_KEY_3>;
        };      
};

longpress {
        compatible = "zephyr,input-longpress";
        input = <&buttons>;
        input-code = <INPUT_KEY_3>;
        short-code = <INPUT_KEY_A>;
        long-codes = <INPUT_KEY_B>, <INPUT_KEY_C>, <INPUT_KEY_D>;
        long-delays-ms = <500>, <1000>, <2000>; 
};

Here the longpress "devices" listens for events from "buttons" and generates more events as needed. A similar approach could be used to convert row-col combinations to keycodes.

The same approach can be used to provide a driver that provides backward compatibility with the current "kscan" API:

chosen {
        zephyr,keyboard-scan = &kscan_adapter;
};

kscan_adapter: kscan-adapter {
        compatible = "zephyr,kscan-adapter";
        input = <&ft5336>;
};

This allows porting the current drivers to the new API, but also to run the existing "kscan" based code unmodified by just defining the adapter device in the board devicetree.

Ultimately, other drivers and subsystems could also use this "sink node" approach to select and configure an input device to get events back into the subsystem. For example, LVGL input configuration could represented as:

/* touchpad or mouse */
lvgl-input-pointer {
        compatible = "zephyr,lvgl-input-pointer";
        input = <&ft5336>;
        invert-x;
        invert-y;
        rotation = <180>;
};

/* encoder with left/right turn and push options */
lvgl-input-encoder {
        compatible = "zephyr,lvgl-input-encoder";
        encoder-input = <&qdec>;
        encoder-input-axis = <INPUT_ABS_Z>;
        push-input = <&buttons>;
        push-input-code = <INPUT_KEY_1>;
};

Proof of concept

A working proof of concept implementation with the subsystem, few drivers, filters and sample is available here: #54620

Development

Feature wise, the initial proposal is going to be just the API with queue or synchronous as an option and required material (samples, documentation, tests).

Followup would be the compatibility adapter, migrating some drivers (gpio-keys, the various EC keyboard scan, ft5336...) out of their current subsystem into a new "drivers/input" directory on the new API and adding some filters as an example.

Ultimately it should be possible to deprecate and drop the kscan API.

Further improvements may include:

• Shell commands for event dumping, debugging, usage stats
• An API to enable/disable the input device (is DEVICE_PM enough for that?)
• Document a suggested way of implementing vendor specific event (so that upstream does not break it), may be enough to define a "INPUT_EV_VND" code

Concerns and Unresolved Questions

• How much devicetree should we use in this? Does it make sense to have commonly used "filters" (longpress, triple click...) represented by a device?
• Should we try to track Linux set of type/code or come up with our own?
• Does it make sense to have embedded sync messages? It saves a message but makes some situation a lot harder, like cycling through keys and syncing on the last one that changed.
• What driver should we convert? Are sensor input devices or sensors (qdec for example).
• How does this interact with userspace?

Alternatives

Keep doing per-device APIs and abuse the current ones. There's enough of those already that PRs are starting to be pushed back with that as a reason.

Use an application specific input system out-of-tree. That's what is currently done already, ZMK for example has an application wide event system which includes keycode state change events but does a lot more and has more features. There's nothing wrong with that and it should be possible for the project to upstream their keyboard and encoder drivers to use the input APIs and then plumb that back in synchronous mode to their own event subsystem with minimal overhead.

An implementation detail worth highlighting is the modes of operation (synchronous or statically allocated queue). Both ZMK and NCS seem to be using dynamic memory in their event implementation. It may be interesting at a later stage to provide a third implementation option that uses dynamically allocated memory for the events, if there's demand for that in some particular use case.

[fabiobaltieri]

cc @petejohanson @gmarull @carlescufi @mbolivar-nordic @nordicjm @galak @dcpleung @tbursztyka (few people that I saw interacting with previous proposals)

[mbolivar-nordic]

It would be nice to see some discussion on the suitability (or lack thereof) of zbus as an event dispatch mechanism for this. cc @rodrigopex

[fabiobaltieri]

It would be nice to see some discussion on the suitability (or lack thereof) of zbus as an event dispatch mechanism for this. cc @rodrigopex

Yeah I did look into that, main blocker from using that is I really wanted to build around iterable sections for listeners (which I think zbus should implement as well). Given that that's implemented, it may be an option, but the core code right now using the k_msgq directly is simple enough I didn't bother too much. It may be worth if we decide that there's a use case for mixing synchronous and asynchronous listeners I guess.

[rodrigopex]

@fabiobaltieri @mbolivar-nordic At first sight, it is possible to zbus to implement the event distribution for the Input Subsystem. I looked at the input.h and input.c, and I could see that part zbus would be helpful.

This change is probably simple (I can help with that), and the listener would be a zbus listener instead of a different listener for inputs. Another topic is whether all the events would be distributed at the same channel or separately based on the configuration files (dts, for example).

The struct input_event can be seen as a message transmitting all the input events or some of that in a channel (or different channels).

struct input_event_msg { 	 
    const struct device *dev; 	 
    uint16_t type; 	 
    uint16_t code; 	 
    int32_t value; 	 
};

I need to go deep to propose something more accurate. However, my first impression is the input subsystem is reimplementing part of the zbus.

Zbus does not use internal message queues to store events. The input_event(...) function can keep pushing to the event queue for being processed by the input thread when INPUT_THREAD is enabled and could publish directly to the channel when it is not. The input thread could consume events from the queue and publish them to the channel.

[rodrigopex]

An implementation detail worth highlighting is the modes of operation (synchronous or statically allocated queue). Both ZMK and NCS seem to be using dynamic memory in their event implementation. It may be interesting at a later stage to provide a third implementation option that uses dynamically allocated memory for the events, if there's demand for that in some particular use case.

Zbus allocates the "events"(messages, in fact) statically by default. However, you can also allocate that dynamically. So I guess zbus solve your needs.

[rodrigopex]

Linux supports input devices using an input event subsystem. Each device registers itself with the system (causing the "/dev/inputX" node to be created among other things), sets the supported capabilities, and then can start sending events whenever available.

You could create a channel for each /dev/inputX with something like zdev_input1 zdev_input2, and so forth. Then, the input subsystem will publish events to the respective channel.

[mbolivar-nordic]

• How much devicetree should we use in this?

IMO as much as you want. Given the complexity of your examples, it seems like the only alternative is a caller-allocated configuration structure that gets passed to the subsystem/driver layers at initialization time, right? How practical is that as an alternative?

Does it make sense to have commonly used "filters" (longpress, triple click...) represented by a device?

I just want to point out that a devicetree node does not have to correspond to a device here, of course, so this is a separate question from whether it should be present in the DT.

[fabiobaltieri]

You could create a channel for each /dev/inputX with something like zdev_input1 zdev_input2, and so forth. Then, the input subsystem will publish events to the respective channel.

Right but then it's more support structure that may not be needed, if we can get away with no device registration (as it is now, anything can just start triggering events with no registration). Let's see where the conversation goes in term of model and features, then we can evaluate if building just on k_msg is enough or we need more.

[fabiobaltieri]

IMO as much as you want. Given the complexity of your examples, it seems like the only alternative is a caller-allocated configuration structure that gets passed to the subsystem/driver layers at initialization time, right? How practical is that as an alternative?

Cool, yeah looking at stuff like io-channels it seems like having more config in devicetree may be the way to go. Mainly concerned with having more struct device allocated for software features but it looks like they don't eat that much flash space these days (24 bytes on my test build? guess getting the PM stuff out helped a lot)

[mkschreder]

I think having things like "longpress" appear as a device tree node is overkill. This can be handled by a generic button driver by simply setting a long press delay property (zero would mean no longpress).

Regarding the way to connect this to the rest of the system, I really like the idea of an input device driver publishing events to a topic and then other drivers subscribing to that topic. Such a mechanism would eliminate a great deal of duplication associated with registering callbacks in other drivers as well.

[fabiobaltieri]

I think having things like "longpress" appear as a device tree node is overkill. This can be handled by a generic button driver by simply setting a long press delay property (zero would mean no longpress).

I think there's value in keeping input driver barebones (generating the minimum amount of events) and having further processing separately, then you could apply the same code to any driver that generates a "button" (imagine touchscreen presses, maybe drivers for capacitive buttons, keyboard scanners etc...). If anything it should be in the application code, but something like this I could see being in the main code base, if anything to have it as a reference.

The alternative would be to have these implemented as a library that the application can link in, that would get rid of the device, it's nice to have the config there though.

[albertofloyd]

@fabiobaltieri is the ultimate goal to deprecate the KSCAN API and convert all the KSCAN drivers in favor of new API?
Or just the ones currently misusing it?

Will the backward compatibility for current applications using KSCAN API be temporary or once this in its final version that option won't be available.

can be used to provide a driver that provides backward compatibility with the current "kscan" API:
Ultimately it should be possible to deprecate and drop the kscan API.