IIS in-process hosting incorrectly throws ConnectionResetException for HTTP POST

[martincostello]

Describe the bug

An ASP.NET Core 2.2.0 application using IIS in-process hosting will intermittently throw ConnectionResetException while reading the request body of an HTTP POST request, causing the request to fail, if the request does not provide an Content-Length HTTP header.

Additionally, if the UseStatusCodePages() exception handler is used, the exception handler fails with an ArgumentException due to a duplicate key.

Based on observed behaviour in our application environment in aggregate, it is possible that this may be related to #6332 and/or #4789 (particularly this).

If a Content-Length HTTP header is specified for the request or if out-of-process hosting is enabled, the exception is not observed.

Stack Trace for exception

Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware:Error: An unhandled exception has occurred while executing the request.

Microsoft.AspNetCore.Connections.ConnectionResetException: The client has disconnected ---> System.IO.EndOfStreamException: Attempted to read past the end of the stream.
   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.Server.IIS.Core.IO.AsyncIOOperation.GetResult(Int16 token)
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.ReadBody()
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.GetReadAsyncResult()
   at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token)
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.ReadAsync(Memory`1 memory, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.IIS.Core.HttpRequestStream.ReadAsyncInternal(Memory`1 buffer, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool`1 bytePool, Nullable`1 limit, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Mvc.Formatters.JsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding)
   at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext bindingContext)
   at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, Object value)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)

Stack trace when status code pages fails

Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware:Error: An exception was thrown attempting to execute the error handler.

System.ArgumentException: An item with the same key has already been added. Key: Allow
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at Microsoft.AspNetCore.HttpSys.Internal.HeaderCollection.Add(String key, StringValues value)
   at Microsoft.AspNetCore.Routing.Matching.HttpMethodMatcherPolicy.<>c__DisplayClass12_0.<CreateRejectionEndpoint>b__0(HttpContext context)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Builder.StatusCodePagesExtensions.<>c__DisplayClass6_0.<<UseStatusCodePagesWithReExecute>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)
Exception thrown: 'Microsoft.AspNetCore.Connections.ConnectionResetException' in System.Private.CoreLib.dll

Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer:Error: Connection ID "18086456109425491975", Request ID "80002fc9-0001-fb00-b63f-84710c7967bb": An unhandled exception was thrown by the application.

To Reproduce

A repro application with instructions can be found here: https://github.com/martincostello/AspNetCoreReproFor6415

Expected behavior

The application correctly responds to the HTTP POST request.

[martincostello]

Have opened #6416 to fix the duplicate key exception.

[pakrym]

I'll take a look on Monday.

[martincostello]

Having a bit a dig into the code myself in Visual Studio with the repro, it looks like the code is reaching this point with a hr value of -2147024858 (HResult 0x80070026) and a bytes value of 0.

https://github.com/aspnet/AspNetCore/blob/3477daf3c4f530dff80f197e0642cb39a26fb07b/src/Servers/IIS/IIS/src/Core/IO/AsyncIOOperation.cs#L108

I guess that points towards the unmanaged part of the code?

[martincostello]

Managed and native call stack is:

>	Microsoft.AspNetCore.Server.IIS.dll!Microsoft.AspNetCore.Server.IIS.Core.IO.AsyncIOOperation.Complete(int hr, int bytes) Line 108	C#
 	Microsoft.AspNetCore.Server.IIS.dll!Microsoft.AspNetCore.Server.IIS.Core.IO.AsyncIOEngine.NotifyCompletion(int hr, int bytes) Line 112	C#
 	Microsoft.AspNetCore.Server.IIS.dll!Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer.OnAsyncCompletion(System.IntPtr pvManagedHttpContext, int hr, int bytes) Line 225	C#
 	[Native to Managed Transition]	
 	aspnetcorev2_inprocess.dll!00007ffb8bbfac54()	Unknown
 	aspnetcorev2_inprocess.dll!00007ffb8bbf166e()	Unknown
 	aspnetcorev2.dll!00007ffbbdbcd9b5()	Unknown
 	iiscore.dll!00007ffbc80e20dd()	Unknown
 	iiscore.dll!00007ffbc80e13f7()	Unknown
 	iiscore.dll!00007ffbc80cea14()	Unknown
 	iiscore.dll!00007ffbc80e2dbd()	Unknown
 	iiscore.dll!00007ffbc80d3789()	Unknown
 	iiscore.dll!00007ffbc80d3e3d()	Unknown
 	w3dt.dll!00007ffbda3f59aa()	Unknown
 	w3dt.dll!00007ffbda3f64d0()	Unknown
 	w3dt.dll!00007ffbda3f4e42()	Unknown
 	w3tp.dll!00007ffbda482bbe()	Unknown
 	w3tp.dll!00007ffbda481819()	Unknown
 	kernel32.dll!BaseThreadInitThunk�()	Unknown
 	ntdll.dll!RtlUserThreadStart�()	Unknown

[JamesNK]

Additionally, if the UseStatusCodePages() exception handler is used, the exception handler fails with an ArgumentException due to a duplicate key.

This is an issue in the new routing HTTP method matcher when an errored request is re-executed. Thanks @martincostello for the PR fix - #6416

If this issue is blocking you then you can use the older routing system in 2.2 by disabling endpoint routing in Startup.cs:

services.UseMvc(options => options.EnableEndpointRouting = false);

// @rynowak

[martincostello]

I only stumbled across the duplicate key exception while trying to write the repro for the connection disconnected exception, but that's useful to know. 👍

[martincostello]

@JamesNK Unrelated to the duplicate key bug, we've just had to disable endpoint routing in an application using status code pages as it was returning 405s to a client application instead of 404s.

Is there something specific required as part of the migration from 2.1 to 2.2 we might have missed that's causing that, or is that a different bug?

Originally we'd disabled endpoint routing because of #4468, but today we refactored the code to not be dependent on the original 2.1 behaviour, so re-enabled endpoint routing.

[pakrym]

I guess that points towards the unmanaged part of the code?

Yes, ConnectionResetException is always caused by IIS signalling about premature request body end.

[pakrym]

There are some traces of the similar issues around (https://blogs.msdn.microsoft.com/napegadie_kones_msft_blog/2016/01/26/502-web-server-received-an-invalid-response-while-acting-as-a-gateway-or-proxy-server/) and it seems to be connected to trying to read request body multiple times.

[pakrym]

I can easily repro the error by starting the client, stopping it and then restarting again (which is semi-expected). But not by just running the client continuously

> dotnet run
Host URL           : http://localhost:50692
Buffer request body: False
........................................................................................................................................................................................................................................................................................................................................................
448722 requests made successfully in 00:06:02.4055324.

[martincostello]

Interesting - I was able to consistently repro it within 1-2 minutes as long as it wasn't buffered.

[pakrym]

It is still a bit strange that request is aborted only after client is restarted so I'll keep looking into this.