New HTTP API implementation

[tgorochowik]

Introduction

This issue is a followup of the Zephyr Networking Forum discussion from March 2019.

Problem description

Since the net_app API has been removed, Zephyr does not have any HTTP APIs.

This should be brought back - this time with the usage of the regular sockets.

Some use cases:

HTTP Server:

• Simple On device web server
• Enabling Websocket API

HTTP Client:

• Simple POST requests to send sensor values
• Enabling Webscocket API

HTTP Authentication:

• At least digest auth supported

Proposed change

As it was mentioned in various places by various people (github, mailing lists), it would be a nice thing to port an existing HTTP implementation, rather than implementing it from scratch.

There are two main things to take into account when researching the libraries:

• Licensing
• It's completeness

Notes on the second bullet: it should be complete, but the features needed for Zephyr need to be evaluated. This library needs to fit to a resource constrained system after all. On the other hand it cannot be so small that it would be equally effective to just write it from scratch.

For example, libcurl was suggested in #13212, as it is considered to be the de-facto standard for HTTP.
The issue with it is: GPL licensed (it is not, I was wrong in this case), and its meant to be complete - i.e. such libs may just be too large for Zephyr

Detailed RFC

We did some research on that, and unfortunately it turns out that there are similar issues with most of what's available. It's either copyleft licensed or extensively large (or both).

What we think could be a good start, is civetweb (https://github.com/civetweb/civetweb).

Some highlights:

• MIT licensed, could be imported to Zephyr
• it is meant to be embeddedable (to other applications - can be a single binary easily)
• it seems to have a well defined API
• seems to be quite flexible (many of its features can be just disabled and compiled out)

Major disadvantages (concerns raised during the Networking Forum):

• Not really ported to any embedded platform
• It seems to be a little bit of everything

That is really something that is done by-design by the developers (it was needed to create an easily embeddable code). Yet this needs to be evaluated whether this is something that is acceptable in Zephyr (as this is something that will have to be dealt with).

To sum up: this is not a perfect candidate, but yet it may be the best there is.

Concerns and Unresolved Questions

The suggested library does not support any client functionality (except a WS client). It is only a HTTP server.

The HTTP client API candidate is still needed.

Alternatives

The very opposite thing - so writing this from scratch. This is something that should probably be taken into account too.

Notes

We are hoping this issue will trigger some discussion and gather opinions. Both on the proposed server API (civetweb), but also on other available libraries that could be good candidates. Opinions about the general idea of porting an existing API instead of writing it from scratch are very welcome too.

[jukkar]

The suggested library does not support any client functionality (except a WS client). It is only a HTTP server.

The HTTP client API candidate is still needed.

In the civetweb README.md file it says HTTP client capable of sending arbitrary HTTP/HTTPS requests. so perhaps the client functionality is hidden there somewhere after all.

[tgorochowik]

You are actually right, there are no examples for a client in the examples dir, but there is something in the main file: https://github.com/civetweb/civetweb/blob/master/src/main.c#L1054
It may be worth to investigate the client capabilities of it too.

[markusr]

There is also Libwebsockets (LWS) which has a http/1.x, http/2 and websockets support for both client and server.

The license is LGPL2 + Static Linking Exception which I think is compatible (IANAL).

[markusr]

On the license of Libwebsockets, see this thread here.

[tgorochowik]

@jukkar A quick update on the client side for civetweb: it is there, and it just works:

$ ./output/build/src/civetweb -C http://localhost
Protocol: http
Host: localhost
Port: 80
Resource: 
Connected to localhost
Response info: 200 OK
<html>
  <head>
    <title>Test File</title>
  </head>
  <body>
    Zephyr!
  </body>
</html>
Closing connection to localhost

(The other side was python -m SimpleHTTPServer 80 in this case).

@markusr The way I understand this is LGPL can generally be used with Apache licensed code, but it cannot really be integrated into Apache licensed projects.

Putting the code into ext seems like a plausible workaround but I honestly don't think it is enough. Please correct me if I am wrong here.

[markusr]

@tgorochowik I don't know if putting the code in ext would be allowed. Who can decide on this?

Here is a comparison table. Please correct and extend.

Feature	civetweb	Libwebsockets (LWS)
License	MIT	LGPL2 + Static Linking Exception
http/1.x client	supported	supported
http/1.x server	supported	supported
http/2.x client	not supported	supported
http/2.x server	not supported	supported
websockets client	supported	supported
websockets server	supported	supported

(I don't have a strong opinion on Libwebsockets I just wanted to add it as an option.)

[jukkar]

We have now support for external modules which means that we do not have to merge external code to ext/ directory, so in this respect using libwebsockets might be possible. But if we would need to modify libwebsocket in any way for zephyr, then those modification would need to be merged into upstream lws sources as otherwise the static linking exception does not apply (if I am reading the license text correctly).

[tgorochowik]

If there is a way to use it, and having subsystems in separate repositories is the way Zephyr wants to go, then personally I have nothing against it either.

Libwebsockets actually looks very promising.

The only issue I see with this approach is using current Zephyrs building blocks vs external building blocks. For example: Zephyr has its own TLS implementation, libwebsockets has its own too. In that way the integration may quite cumbersome. Integrating the lib into Zephyr has its own disadvantages, but I think it makes it much easier to make a fully tailored solution which is quite important here.

[chinli]

@tgorochowik how's going about the http api. Is there any alternative solution?

[tgorochowik]

@chinli Hopefully at least a minimal HTTP API implementation will be ready for 1.15. So far we don't even have a decision as you can see.

If by alternative solution you mean something that you can use now and with current Zephyr tree, then I am afraid there may be nothing that will work out of the box.
Implementing/porting something to be a part of the application is always an option though.

You are also very welcome to share your thoughts on what would you like to see in Zephyr in terms of the HTTP API!

[chinli]

@tgorochowik well, so badly. I have no idea about this issue, but if there was any schedule upgrade or need someone work it, please let me know. thanks!

[tgorochowik]

Since civetweb is now available as a west module and there is a PR open for a minimal HTTP API implementation there is no need for this issue to stay open. Closing.