Kestrel rejects requests with non standard ASCII characters

[ytqsl]

Description

When sending a request to Kestrel web server with header value which has non standard ASCII characters, the server rejects the response with 400 Bad request: malformed headers.

Reproduction steps

Steps to reproduce the behavior:

1. create a new web app: dotnet new web
2. run the application as console (not IISExpress)
3. send a request (using Fiddler/postman/wget) with non standard ASCII header value

This is reproducible with 2.1.9, 2.1.10 and 2.2.4

wget -Uri http://localhost:5000 -Headers @{"test"="Tk'emlúps te Secwépemc"}

Expected behavior

The request is accepted and the header value is accessible in the Request.Headers

Additional context

.NET Core SDK (reflecting any global.json):
Version: 2.2.203
Commit: e5bab63eca

Runtime Environment:
OS Name: Windows
OS Version: 10.0.17763
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\2.2.203\

Host (useful for support):
Version: 2.2.4
Commit: f95848e524

.NET Core SDKs installed:
2.1.505 [C:\Program Files\dotnet\sdk]
2.1.603 [C:\Program Files\dotnet\sdk]
2.2.106 [C:\Program Files\dotnet\sdk]
2.2.203 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

[ytqsl]

Forgot to mention that when using IIS/IISExpress, inproc mode accepts the header while the default out of proc doesn't (as 'expected'...)

[blowdart]

@JunTaoLuo / @Tratcher

What is the expected behaviour here? You know I believe rejecting the headers is the correct default, with an option to accept invalid and possibly dangerous input being left as opt-in, for all hosts.

[Tratcher]

UTF-8 sent directly to Kestrel should work. No other encodings are accepted. We'd want a wireshark trace to verify the encoding.

If you send UTF-8 through IIS out-of-proc the encoding gets messed up and rejected.

[Tratcher]

Note the UTF-8 support was added in 2.2 (aspnet/KestrelHttpServer#1144). That's the only version we need to investigate.

[ytqsl]

@Tratcher here's the Wireshark trace. IIS is not involved.

ws capture.zip

[Tratcher]

0050   48 54 54 50 2f 31 2e 31 0d 0a 74 65 73 74 3a 20   HTTP/1.1..test: 
0060   54 6b 27 65 6d 6c fa 70 73 20 74 65 20 53 65 63   Tk'eml.ps te Sec
0070   77 e9 70 65 6d 63 0d 0a 55 73 65 72 2d 41 67 65   w.pemc..User-Age
0080   6e 74 3a 20 4d 6f 7a 69 6c 6c 61 2f 35 2e 30 20   nt: Mozilla/5.0 

That's not UTF-8, it's ANSII. ú is encoded as fa. In UTF-8 it would be \xC3\xBA.

[ytqsl]

I tried the following headers, all didn't return 400, but the value was read as is from ctx.request.headers:

wget -Uri http://localhost:5000 -Headers @{"test"="\x54\x6b\x27\x65\x6d\x6c\xc3\xba\x70\x73\x20\x74\x65\x20\x53\x65\x63\x77\xc3\xa9\x70\x65\x6d\x63"; "Accept-Charset"="UTF-8"}

wget -Uri http://localhost:5000 -Headers @{"test"="\x54\x6b\x27\x65\x6d\x6c\xc3\xba\x70\x73\x20\x74\x65\x20\x53\x65\x63\x77\xc3\xa9\x70\x65\x6d\x63"}

wget -Uri http://localhost:5000 -Headers @{"test"="Tk'eml\xC3\xBAps te Secw\xC3\xBApemc"; "Accept-Charset"="UTF-8"}

[Tratcher]

Wireshark? Are you sure it's not sending the literal string \x54 rather than the byte 54?

[ytqsl]

Any idea how to force sending a byte array instead of a string as the header value?

[Tratcher]

No, very few clients let you control the header encoding.

[ytqsl]

Is there a way to inject custom code before Kestrel rejects the request, or I must have a reverse proxy in front of it?