Introduce option parameter for shields

[soburi]

Introduction

Zephyr's shield system, inspired by the Arduino ecosystem, can integrate expansion boards easily. However, the current framework struggles with boards featuring multiple connectors, such as Grove. This proposal addresses these gaps by extending the shield system to support flexible configurations for boards with multiple connectors.

https://builds.zephyrproject.io/zephyr/pr/82825/docs/hardware/porting/shields.html
https://builds.zephyrproject.io/zephyr/pr/82825/docs/doxygen/html/group__dts__shield__option__apis.html#details

Problem description

Zephyr's existing shield system is designed for single-connector configurations, so it is not flexible. So, supporting multi-connector boards like those used in the Grove ecosystem is challenging. We can't even connect it to a different port. This limitation restricts the use of shields across varying hardware configurations.

Proposed change

This proposal enhances the shield framework to enable flexible configurations at build time. By extending the --shield option in the west build command, users can specify parameters such as I2C addresses or GPIO pins directly. Also, users can use multiple instances of the same shield.

A typical example is when you specify

--shield seeed_grove_lis3dhtr@1:addr=0x18

it provides a mechanism to switch the connection to grove_i2c1.
Options can be freely defined, and determining them in the overlay file provides the flexibility to change the shield definition (slightly) from the outside.

I am creating an implementation at #82825. (Work in progress)

Proposed change (Detailed)

Command-Line Parameterization:

We will enhance the --shield option to accept detailed parameters for individual shield configurations. This lets users specify attributes like connector names, I2C addresses, and GPIO pins directly from the command line. The format enables defining multiple configurations for the same shield, ensuring flexibility and efficiency.

The format will allow multiple options to be passed for a single shield, in the format:

<shield_name>[@<index>][:<option>{=<value>}[:<option>{=<value>}]]

Parameterizable overlay definition

Defines a macro so that options specified on the command line can be reflected in the overlay.

Details are provided in the following document.

https://builds.zephyrproject.io/zephyr/pr/82825/docs/doxygen/html/group__dts__shield__option__apis.html#autotoc_md6

Derived overlay:

Shields need to be configured differently for each option specified. To achieve this, a new "derived overlay" concept will be introduced. The build process creates "derived overlays" internally for each option given on the command line.

The details are described in the document.
https://builds.zephyrproject.io/zephyr/pr/82825/docs/hardware/porting/shields.html#derived-overlay

The actual process is described in the documentation.
https://builds.zephyrproject.io/zephyr/pr/82825/docs/doxygen/html/group__dts__shield__option__apis.html#autotoc_md7

Conclusion

This extension to the Zephyr shield framework provides greater flexibility for users working with multi-connector systems like Grove. By allowing configuration parameters to be specified at build time, it becomes much easier to integrate and customize multiple instances of shields on the same device. The use of SHIELD_BASE_NAME and SHIELD_DERIVED_NAME macros ensures unique configuration names, while default values and the direct referencing of SHIELD_OPTION allow for smooth and efficient configurations even without user input.

The example with GPIO interrupts illustrates how this framework can be adapted to various peripherals, enabling more complex hardware configurations in a simple and flexible manner.

[soburi]

@fabiobaltieri @kartben

I know you are an expert in this domain and would love to hear your thoughts if you have the time.

[fabiobaltieri]

Are we?! Cool! :-) Adding few people I think may have opinions/ideas here

cc @tejlmand @gmarull @bjarki-andreasen @henrikbrixandersen @erwango @decsny

Yeah I think it's a cool idea, some macro and devicetree shenanigans so I think it's a good fit for the project. :-)

This supports multiple instance of the same shield on different busses right? (sorry the cmake is a bit beyond me... weird, I thought I was an expert on it :-)).

Also, should this support changing the address as well? Some of those i2c boards have solder jumpers for address and you can daisy chain them so that may be a useful second option to have multiple of those on the same bus. Maybe even give a more rigid structure to the option? like `--shield <shield_name>@:

or something like that?

Anyway, good stuff.

[bjarki-andreasen]

When working with the nordic WiFi shields, there are two primary issues:

• Having more than connector of the same type, identified by this PR
• Having only a subset of a connector available (busses, pins), like a specific microbBUS connector instance missing SPI or reset PIN.

Having multiple connectors, the simplest way to support it is to index them, and use the --shield foo@0, --shield foo@1 like @fabiobaltieri suggested, and then having the connectors named arduino_i2c0, arduino_i2c1 etc.

The second issue however, if a pin is missing from the connector, but referenced in a shield overlay:
connector:

	arduino_header: connector {
		compatible = "arduino-header-r3";
		#gpio-cells = <2>;
		gpio-map-mask = <0xffffffff 0xffffffc0>;
		gpio-map-pass-thru = <0 0x3f>;
		gpio-map = <0 0 &gpio0 4 0>,	/* A0 */
			   <1 0 &gpio0 5 0>,	/* A1 */
			   <6 0 &gpio1 0 0>,	/* D0 */
			   <7 0 &gpio1 1 0>,	/* D1 */
			   <8 0 &gpio1 4 0>,	/* D2 */
			   <9 0 &gpio1 5 0>,	/* D3 */
	};

shield:

	foo: foo {
		compatible = "vnd,foo";
		status = "okay";

                /* D2 */
		status0-gpios = <&arduino_header 8 GPIO_ACTIVE_HIGH>;
        };

currently, this will fail since when looking up arduino_header 8, its not found. If we try to add a snippet/overlay to remove the reference from the shield:

&foo {
        /delete-node/ status0-gpios;
};

it still fails since the lookup is performed before a snippet or overlay is applied.

We essentially need something like:
shield:

	foo: foo {
		compatible = "vnd,foo";
		status = "okay";

#if CONFIG_ARDUINO_HEADER_8_EXISTS
                /* D2 */
		status0-gpios = <&arduino_header 8 GPIO_ACTIVE_HIGH>;
#endif
        };

to exclude it.

[bjarki-andreasen]

Regarding implementation, I think this would be solved better with a yaml schema applied to the board somewhere, which would be used to generate code using python. something like:

schema:

board:
  connectors:
    - arduino
    - microbus

from which we would look up any nodelabel starting with arduino or microbus:

arduino0_header: connector {
        ...
};
arduino0_spi: &spi4 {
        ...
};

and generate the header from this:

#define SHIELD_ARDUINO0_EXISTS 1
#define SHIELD_ARDUINO0_HAS_SPI 1
#define SHIELD_ARDUINO0_HAS_HEADER 1

which we could easily make support existing boards by automatically appending 0 to the node name if there is only one of them, so arduino_header would match SHIELD_ARDUINO0_EXISTS 1.

This would lastly be able to fix the individual pin issue as well if we add pin awareness to the yaml schema:

board:
  connectors:
    - arduino:
      header:
        gpios: 16

which would allow the python script to check the gpio-map for existance of pins 0, 1, 2, ... and generate:

#define SHIELD_ARDUINO0_HAS_HEADER_GPIOS_0 1
#define SHIELD_ARDUINO0_HAS_HEADER_GPIOS_1 0
#define SHIELD_ARDUINO0_HAS_HEADER_GPIOS_2 1

This implementation is really flexible and should be easier readable than cmake (only a few wizards know that tung :D)

[soburi]

@fabiobaltieri @bjarki-andreasen

I'm sorry for not getting back to you sooner.

I updated the PR to reflect your suggestion.
I haven't implemented any tests yet, but it seems to work well as a proof of concept.

This supports multiple instance of the same shield on different busses right? (sorry the cmake is a bit beyond me... weird, I thought I was an expert on it :-)).

After some consideration, we were able to achieve this.
However, I had to change the build mechanism, so I want you to review this.

It is described in https://builds.zephyrproject.io/zephyr/pr/82825/docs/hardware/porting/shields.html#derived-overlay

Also, should this support changing the address as well? Some of those i2c boards have solder jumpers for address and you can daisy chain them so that may be a useful second option to have multiple of those on the same bus.

This was solved by having the data internally in a generic key=value format.
Options can be arbitrarily defined using a schema, providing a comprehensive solution.

https://builds.zephyrproject.io/zephyr/pr/82825/docs/hardware/porting/shields.html#specifying-parameters

Maybe even give a more rigid structure to the option? like `--shield <shield_name>@:

Having multiple connectors, the simplest way to support it is to index them, and use the --shield foo@0, --shield foo@1 like @fabiobaltieri suggested, and then having the connectors named arduino_i2c0, arduino_i2c1 etc.

The "@" notation is also supported.
As part of general-purpose mechanism, it is treated as just a "number." The shield developer can decide how to use this number, but it will generally converge to the "connector number."

This implementation is really flexible and should be easier readable than cmake (only a few wizards know that tung :D)

Pythonization has been addressed.👍

Regarding implementation, I think this would be solved better with a yaml schema applied to the board somewhere, which would be used to generate code using python. something like:

The shield.yml has also been supported. However, as described later, some suggestions for @... seemed difficult to implement, so it may be a little different from what you expected.

https://builds.zephyrproject.io/zephyr/pr/82825/docs/hardware/porting/shields.html#shield-yaml-format

Adaptive configuration

Now that we can define flexible options,
it's possible to switch between the following cases with #ifdef on the command line.
I can say this is a small step forward.

	foo: foo {
		compatible = "vnd,foo";
		status = "okay";

#if CONFIG_ARDUINO_HEADER_8_EXISTS
                /* D2 */
		status0-gpios = <&arduino_header 8 GPIO_ACTIVE_HIGH>;
#endif
        };

But,

#82889 (comment)

which would allow the python script to check the gpio-map for existance of pins 0, 1, 2, ... and generate:

I think the @bjarki-andreasen proposed is a bit challenging to implement now.

The processing of dts is roughly as follows.

1. Preprocessing
2. Creating EDT Creating EDT.pickle (gen_edt.py)
3. Generating devicetree-generated.h (gen_defines.py)
4. Normal build of C source

To analyze DTS, all preprocessor conditions must be resolved, and this approach reverses the order. (Bootstrap problem!)
In other words, macro generation and resolution must be completed at point 1, and with the current mechanism, I think it is impossible to handle cross-references between device trees.

Based on the @bjarki-andreasen proposed idea,
if we extend the board schema to include the shield definition,
it would be possible to automatically generate files like arduino_r3_connector.dtsi from this schema.
This means that duplicate definitions can be avoided.
With this method, you can get the configuration information without going through DTS, so all the processing can be done at step 1 mentioned above.
Alternatively, it would be good to consider a mechanism to ensure that the schema and DTS do not diverge, but I think this would be difficult because DTS is a flexible mechanism.

This proposal is also large enough, so it is good to discuss a separate issue.

[decsny]

This proposal seems to have matured pretty well so far, should it be discussed in architecture review ? @henrikbrixandersen @carlescufi

[henrikbrixandersen]

This proposal seems to have matured pretty well so far, should it be discussed in architecture review ? @henrikbrixandersen @carlescufi

@soburi Will you be able to present this to the Arch WG meeting, perhaps next week?