HTTPs and "Connection: close" - huge performance degradation on windows

[surr34]

We have a Kestrel service that recently receives a lot of https requests containing a Connection: close header. It seems the header has a massive impact on the number of requests we can handle. Testing with a simple service locally, the RPS drops from over 30k to under 500.

For testing purpose we ran the same service on Linux and implemented it in NodeJS as well. While the C# version on Windows was fast for kept-alive connections, the performance degradation was huge. Using Connection: close, the C# Windows version was outperformed by both NodeJS and the C# implementation on a Linux system. Based on other Github issues I believed the Windows version to be way faster than the Linux one. However below the number (RPS) we measured

	keep-alive	close
Kestrel (win)	54641	349
Node (win)	10422	2256
Kestrel (Linux)	51490	2339
Node (Linux)	37504	2095

For the Kestrel Windows version we also noticed a significant spike in the CPU consumption of the lsass process.

[stephentoub]

Since this is focused on Kestrel, I've moved it for now to dotnet/aspnetcore. If investigation demonstrates it's instead due to something lower in the stack, we can move it back. Thanks.

[Tratcher]

With or without TLS (HTTPS)? lsass implies TLS.

[surr34]

@Tratcher Tests were run with TLS (HTTPS)

[Tratcher]

@stephentoub @halter73 haven't we seen other reports of SslStream handshakes being extremely expensive?

[analogrelay]

We can do a quick investigation with some timers to confirm, but I'm pretty sure it's the SslStream handshake issues we've seen for a while (@karelz you're already tracking SSL performance improvements, right?).

[karelz]

@anurse we track SSL perf, but more for read/write throughput. While I've heard about handshake perf several times, I didn't see any data or anything.
Do we have something like that to help us focus?

[analogrelay]

Yep, we'll try to do a quick bit of work to isolate what we can. If I remember correctly, the biggest signal we have is our benchmarks show a significant performance gap between ConnectionClose and ConnectionClose with HTTPS.

Improving HTTPS perf in general is on our radar, so this is an area we'll be working on.

[samsosa]

Maybe the pull request dotnet/runtime#1949 helps.