net: Regression in multiple client connection handling with samples/net/sockets/dumb_http_server (frdm_k64f/Ethernet)

[pfalcon]

(Split off from #7365 (comment))

One of the basic functional/regression tests for the IP stack is issuing 1K of HTTP requests to the samples/net/sockets/dumb_http_server sample. In 1.10, it works as expected (serving this 1K of requests).

But with the current master, running on frdm_k64f with Ethernet, it's:

$ ab -n1000 http://192.0.2.1:8080/
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.0.2.1 (be patient)
apr_socket_recv: Connection reset by peer (104)
Total of 3 requests completed

[pfalcon]

The offending commit is a0371ae, #5442

@rveerama1 : FYI.

[pfalcon]

Investigating the issue, it is by-product of the #5442 implementation. We used to have Kconfig setting of CONFIG_NET_TCP_TIME_WAIT, description "Enable TCP TIME_WAIT timeouts", default "n". Apparently, it implemented only partial (if at all) support for TIME_WAIT state. In #5442, TIME_WAIT was implemented, and existing config vars were ignored, so the new TIME_WAIT implementation is always on.

And the expected outcome of TIME_WAIT is keeping existing contexts/connections allocated for some time even after they're formally closed. So, attempt to allocate a new "accepted" context leads to error, which is handled by RSTing the client connection. There're 2 issues here:

1. Handling of client connection SYN sequence in Zephyr is sub-ideal. We always reply to SYN with SYN+ACK, and only later when receive ACK from peer, check whether we actually have resources to create a connection. If not, we reset it in peer's face. The behavior of "Apache Bench" can be taken as a reference one - it treats such error as a fatal in the server, meaning that it's unable to pass the basic load test. A better way to handle the situation above would be not to respond to SYN, if we don't have enough resources for connection. Then, the peer would retransmit SYN few times, before giving up, and overall handling of connections and resources will be more smooth, scalable, and friendly. However, any changes like that can affect DoSability of the IP stack, so require additional consideration. I don't have immediate plans to do anything on that regard, but will probably create a separate ticket for this issue.
2. After Handle properly TCP connection closure #5442, it's impossible to disable TIME_WAIT state (which is not friendly to low-memory server applications). For that, net: Multiple inconsistent settings to configure TIME_WAIT delay in the IP stack #7459 is created, and net: tcp: Define single config option for TIME_WAIT delay #7476 patch submitted. With that patch and CONFIG_NET_TCP_TIME_WAIT_DELAY=0, the issue in this ticket is resolved.