RFC: Requirements for implementations of high(er)-level networking protocols in Zephyr (wip) #13212
Description
Introduction
This RFC proposes a set of requirements and guidelines to design API structure, contracts, and behavior for application-level network protocols in Zephyr.
Problem description
Zephyr has some history (arguably, dramatic history) of networking in general, and implementations for application-level protocols in particular. Older, net-app based protocol implementation had various reported issues and criticism, and eventually were removed completely together with net-app itself.
Starting with Sockets API as a baseline allows us to design new implementations to address previously known issues, and at the same time, to be consistent and compatible among themselves. We should pursue that opportunity.
Summary
Application-level protocols should:
- Not impose arbitrary restrictions on the amount of data processed (as an none of general-purpose networking protocols imposes such arbitrary restrictions).
- May impose arbitrary restrictions on metadata size, but should try to make those restrictions/limits configurable (at compile time).
- Should try to implement basic features of the protocol, followed by more advanced, in that order, not leaving random gaps in the functionality.
- Should follow the general structure and signatures of the underlying API, as a means of design consistency and self-organization.
- Should strive for composabilty (largely enabled by the previous point).
Detailed RFC
Note: that RFC largely focuses on TCP (SOCK_STREAM) protocols, as construing majority of protocols used in the Internet. However, many ideas apply in a natural way to UDP protocols too.
Definitions:
- data - user data as transferred by protocols.
- metadata - various auxiliary, usually structured, information describing user data.
Examples: In HTTP, request and response payload is data, while request and response headers are metadata. In MQTT, message payload is data, while message headers, topic names, etc. are metadata. In WebSocket, message payload is data, while headers are metadata. Etc.
Detailed requirements:
- Size of data handled by the protocol implementations should not have arbitrary limits. For protocols with sequential access (majority of protocols), this should also include not being limited by the amount of RAM available. This necessitates support of the streaming of data, where user is able to process/generate data incrementally.
- Streaming support for metadata would generally make API too complex. Thus, it's it's considered OK to: a) limit amount of metadata to the amount fitting in RAM; b) beyond that, for efficiency of implementation, further limits may be imposed, e.g. to allow to pre-allocate static buffers to avoid dynamic allocation and failure modes involved with it; c) but, whenever possible such limits should be configurable using Kconfig.
Examples of the applications of these requirements:
- HTTP doesn't impose arbitrary limits on the payload size, so neither Zephyr libs should. Instead, they should allow to read/write payload chunk by chunk. On the other hand, for simplification of API, we may impose limits on the request/response headers, e.g. say that we handle max 1KB of headers (and allow to change that via Kconfig). If there're good reasons for that, we may also say that a single header is limited to 200 bytes. And that header name is limited to 64 bytes.
- MQTT allows for payloads of up to 256MB. Zephyr implementation should not impose lower limits. Again, this is only achievable via streaming. MQTT allows for topic names up to 64K. We may by default impose lower limit, and allow to configure that via Kconfig.
- As set forth in the previous point, streaming support is the baseline requirement. The way to implement it should be providing functions which exactly follow the signature of the POSIX read() and write() functions, modulo the type of the first argument. Specifically, to read and write payload in chunks, protocol library should provide functions:
- ssize_t xxx_read_payload(client_ctx *ctx, void *buf, size_t size)
- ssize_t xxx_write_payload(client_ctx *ctx, const void *buf, size_t size)
(Only signature is important, this RFC doesn't seek to stipulate API names ahead of concrete implementation, though standardizing on them would be a bonus). These signatures should not be augmented with any other parameters (at least, for protocols which deal with unstructured (or homogenuous) data, which is usually the case).
- For when additional parameters should be provided to protocol library, they likewise should be handled in a manner similar to sockets - using dedicated param getter/setter functions, similar to getsockopt/setsockopt (also to fcntl/ioctl).
- Two previous requirements can be summarized as: "API principles for higher-level protocols should mimic that of the underlying API, as means of self-documenting and self-policing (avoiding complicated, inconsistent, adhoc APIs)". Note that on top of these generic APIs, which are mandatory, protocol-specific convenience API may be defined, to ease usage in simplified scenarios.
Proposed change (Detailed)
This section is freeform - you should describe your change in as much detail
as possible. Please also ensure to include any context or background info here.
For example, do we have existing components which can be reused or altered.
By reading this section, each team member should be able to know what exactly
you're planning to change and how.
Dependencies
Highlight how the change may affect the rest of the project (new components,
modifications in other areas), or other teams/projects.
Concerns and Unresolved Questions
List any concerns, unknowns, and generally unresolved questions etc.
Alternatives
List any alternatives considered, and the reasons for choosing this option
over them.