HttpSys test failures on 20145.1001

[Tratcher]

A number of HttpSys tests rely on new features only available in later Win10 versions. We don't have any agents with current or future Win10 versions so we don't see when these tests regress (https://github.com/dotnet/aspnetcore-internal/issues/3170).

When running HttpSys tests locally for another issue I saw the following failures:

Windows version 2004 (20145.1001) - Insider Slow?

Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.AppException_AfterHeaders_ResetInternalError Failed	312 ms		System.AggregateException : One or more hosted services failed to stop. (Assert.Equal('RST_STREAM', 'DATA') failed) ---- System.Exception : Assert.Equal('RST_STREAM', 'DATA') failed
Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.AppException_BeforeHeaders_500 Failed	23 ms		System.AggregateException : One or more hosted services failed to stop. (Assert.Equal('END_STREAM', 'NONE') failed) ---- System.Exception : Assert.Equal('END_STREAM', 'NONE') failed
Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.ConnectionClose_AdditionalRequests_ReceivesSecondGoAway Failed	22 ms		System.AggregateException : One or more hosted services failed to stop. (Assert.Equal('END_STREAM', 'NONE') failed) ---- System.Exception : Assert.Equal('END_STREAM', 'NONE') failed
Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.ConnectionClose_OSSupport_SendsGoAway Failed	20 ms		System.AggregateException : One or more hosted services failed to stop. (Assert.Equal('END_STREAM', 'NONE') failed) ---- System.Exception : Assert.Equal('END_STREAM', 'NONE') failed
Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.Reset_AfterResponseHeaders_Resets Failed	17 ms		System.AggregateException : One or more hosted services failed to stop. (Assert.Equal('RST_STREAM', 'DATA') failed) ---- System.Exception : Assert.Equal('RST_STREAM', 'DATA') failed
Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.Reset_CompleteAsyncDurringRequestBody_Resets Failed	32 ms		System.AggregateException : One or more hosted services failed to stop. (Assert.Equal('END_STREAM', 'NONE') failed) ---- System.Exception : Assert.Equal('END_STREAM', 'NONE') failed

Test Name:	Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.AppException_AfterHeaders_ResetInternalError
Test FullName:	Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.AppException_AfterHeaders_ResetInternalError
Test Source:	D:\github\AspNetCore\src\Servers\HttpSys\test\FunctionalTests\Http2Tests.cs : line 247
Test Outcome:	Failed
Test Duration:	0:00:00

Test Name:	Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.AppException_AfterHeaders_ResetInternalError
Test Outcome:	Failed
Result StackTrace:	
at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.WaitForShutdownAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.AppException_AfterHeaders_ResetInternalError() in D:\github\AspNetCore\src\Servers\HttpSys\test\FunctionalTests\Http2Tests.cs:line 255
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
   at Microsoft.AspNetCore.Http2Cat.Http2Utilities.Assert.Equal[T](T expected, T actual) in D:\github\AspNetCore\src\Shared\Http2cat\Http2Utilities.cs:line 1059
   at Microsoft.AspNetCore.Http2Cat.Http2Utilities.VerifyResetFrame(Http2Frame frame, Int32 expectedStreamId, Http2ErrorCode expectedErrorCode) in D:\github\AspNetCore\src\Shared\Http2cat\Http2Utilities.cs:line 982
   at Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.<>c.<<AppException_AfterHeaders_ResetInternalError>b__5_1>d.MoveNext() in D:\github\AspNetCore\src\Servers\HttpSys\test\FunctionalTests\Http2Tests.cs:line 270
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Http2Cat.Http2CatHostedService.RunAsync() in D:\github\AspNetCore\src\Shared\Http2cat\Http2CatHostedService.cs:line 105
   at Microsoft.AspNetCore.Http2Cat.Http2CatHostedService.RunAsync() in D:\github\AspNetCore\src\Shared\Http2cat\Http2CatHostedService.cs:line 121
   at Microsoft.AspNetCore.Http2Cat.Http2CatHostedService.RunAsync() in D:\github\AspNetCore\src\Shared\Http2cat\Http2CatHostedService.cs:line 121
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
Result Message:	
System.AggregateException : One or more hosted services failed to stop. (Assert.Equal('RST_STREAM', 'DATA') failed)
---- System.Exception : Assert.Equal('RST_STREAM', 'DATA') failed

Test Name:	Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.AppException_BeforeHeaders_500
Test FullName:	Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.AppException_BeforeHeaders_500
Test Source:	D:\github\AspNetCore\src\Servers\HttpSys\test\FunctionalTests\Http2Tests.cs : line 183
Test Outcome:	Failed
Test Duration:	0:00:00

Test Name:	Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.AppException_BeforeHeaders_500
Test Outcome:	Failed
Result StackTrace:	
at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.WaitForShutdownAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.AppException_BeforeHeaders_500() in D:\github\AspNetCore\src\Servers\HttpSys\test\FunctionalTests\Http2Tests.cs:line 190
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
   at Microsoft.AspNetCore.Http2Cat.Http2Utilities.Assert.Equal[T](T expected, T actual) in D:\github\AspNetCore\src\Shared\Http2cat\Http2Utilities.cs:line 1059
   at Microsoft.AspNetCore.Http2Cat.Http2Utilities.VerifyDataFrame(Http2Frame frame, Int32 expectedStreamId, Boolean endOfStream, Int32 length) in D:\github\AspNetCore\src\Shared\Http2cat\Http2Utilities.cs:line 966
   at Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.<>c.<<AppException_BeforeHeaders_500>b__3_1>d.MoveNext() in D:\github\AspNetCore\src\Servers\HttpSys\test\FunctionalTests\Http2Tests.cs:line 205
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Http2Cat.Http2CatHostedService.RunAsync() in D:\github\AspNetCore\src\Shared\Http2cat\Http2CatHostedService.cs:line 105
   at Microsoft.AspNetCore.Http2Cat.Http2CatHostedService.RunAsync() in D:\github\AspNetCore\src\Shared\Http2cat\Http2CatHostedService.cs:line 121
   at Microsoft.AspNetCore.Http2Cat.Http2CatHostedService.RunAsync() in D:\github\AspNetCore\src\Shared\Http2cat\Http2CatHostedService.cs:line 121
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
Result Message:	
System.AggregateException : One or more hosted services failed to stop. (Assert.Equal('END_STREAM', 'NONE') failed)
---- System.Exception : Assert.Equal('END_STREAM', 'NONE') failed

Test Name:	Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.ConnectionClose_AdditionalRequests_ReceivesSecondGoAway
Test FullName:	Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.ConnectionClose_AdditionalRequests_ReceivesSecondGoAway
Test Source:	D:\github\AspNetCore\src\Servers\HttpSys\test\FunctionalTests\Http2Tests.cs : line 109
Test Outcome:	Failed
Test Duration:	0:00:00

Test Name:	Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.ConnectionClose_AdditionalRequests_ReceivesSecondGoAway
Test Outcome:	Failed
Result StackTrace:	
at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.WaitForShutdownAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.ConnectionClose_AdditionalRequests_ReceivesSecondGoAway() in D:\github\AspNetCore\src\Servers\HttpSys\test\FunctionalTests\Http2Tests.cs:line 117
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
   at Microsoft.AspNetCore.Http2Cat.Http2Utilities.Assert.Equal[T](T expected, T actual) in D:\github\AspNetCore\src\Shared\Http2cat\Http2Utilities.cs:line 1059
   at Microsoft.AspNetCore.Http2Cat.Http2Utilities.VerifyDataFrame(Http2Frame frame, Int32 expectedStreamId, Boolean endOfStream, Int32 length) in D:\github\AspNetCore\src\Shared\Http2cat\Http2Utilities.cs:line 966
   at Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.<>c.<<ConnectionClose_AdditionalRequests_ReceivesSecondGoAway>b__2_1>d.MoveNext() in D:\github\AspNetCore\src\Servers\HttpSys\test\FunctionalTests\Http2Tests.cs:line 138
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Http2Cat.Http2CatHostedService.RunAsync() in D:\github\AspNetCore\src\Shared\Http2cat\Http2CatHostedService.cs:line 105
   at Microsoft.AspNetCore.Http2Cat.Http2CatHostedService.RunAsync() in D:\github\AspNetCore\src\Shared\Http2cat\Http2CatHostedService.cs:line 121
   at Microsoft.AspNetCore.Http2Cat.Http2CatHostedService.RunAsync() in D:\github\AspNetCore\src\Shared\Http2cat\Http2CatHostedService.cs:line 121
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
Result Message:	
System.AggregateException : One or more hosted services failed to stop. (Assert.Equal('END_STREAM', 'NONE') failed)
---- System.Exception : Assert.Equal('END_STREAM', 'NONE') failed

Test Name:	Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.ConnectionClose_OSSupport_SendsGoAway
Test FullName:	Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.ConnectionClose_OSSupport_SendsGoAway
Test Source:	D:\github\AspNetCore\src\Servers\HttpSys\test\FunctionalTests\Http2Tests.cs : line 70
Test Outcome:	Failed
Test Duration:	0:00:00

Test Name:	Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.ConnectionClose_OSSupport_SendsGoAway
Test Outcome:	Failed
Result StackTrace:	
at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.WaitForShutdownAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.ConnectionClose_OSSupport_SendsGoAway() in D:\github\AspNetCore\src\Servers\HttpSys\test\FunctionalTests\Http2Tests.cs:line 78
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
   at Microsoft.AspNetCore.Http2Cat.Http2Utilities.Assert.Equal[T](T expected, T actual) in D:\github\AspNetCore\src\Shared\Http2cat\Http2Utilities.cs:line 1059
   at Microsoft.AspNetCore.Http2Cat.Http2Utilities.VerifyDataFrame(Http2Frame frame, Int32 expectedStreamId, Boolean endOfStream, Int32 length) in D:\github\AspNetCore\src\Shared\Http2cat\Http2Utilities.cs:line 966
   at Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.<>c.<<ConnectionClose_OSSupport_SendsGoAway>b__1_1>d.MoveNext() in D:\github\AspNetCore\src\Servers\HttpSys\test\FunctionalTests\Http2Tests.cs:line 98
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Http2Cat.Http2CatHostedService.RunAsync() in D:\github\AspNetCore\src\Shared\Http2cat\Http2CatHostedService.cs:line 105
   at Microsoft.AspNetCore.Http2Cat.Http2CatHostedService.RunAsync() in D:\github\AspNetCore\src\Shared\Http2cat\Http2CatHostedService.cs:line 121
   at Microsoft.AspNetCore.Http2Cat.Http2CatHostedService.RunAsync() in D:\github\AspNetCore\src\Shared\Http2cat\Http2CatHostedService.cs:line 121
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
Result Message:	
System.AggregateException : One or more hosted services failed to stop. (Assert.Equal('END_STREAM', 'NONE') failed)
---- System.Exception : Assert.Equal('END_STREAM', 'NONE') failed

Test Name:	Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.Reset_AfterResponseHeaders_Resets
Test FullName:	Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.Reset_AfterResponseHeaders_Resets
Test Source:	D:\github\AspNetCore\src\Servers\HttpSys\test\FunctionalTests\Http2Tests.cs : line 361
Test Outcome:	Failed
Test Duration:	0:00:00

Test Name:	Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.Reset_AfterResponseHeaders_Resets
Test Outcome:	Failed
Result StackTrace:	
at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.WaitForShutdownAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.Reset_AfterResponseHeaders_Resets() in D:\github\AspNetCore\src\Servers\HttpSys\test\FunctionalTests\Http2Tests.cs:line 381
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
   at Microsoft.AspNetCore.Http2Cat.Http2Utilities.Assert.Equal[T](T expected, T actual) in D:\github\AspNetCore\src\Shared\Http2cat\Http2Utilities.cs:line 1059
   at Microsoft.AspNetCore.Http2Cat.Http2Utilities.VerifyResetFrame(Http2Frame frame, Int32 expectedStreamId, Http2ErrorCode expectedErrorCode) in D:\github\AspNetCore\src\Shared\Http2cat\Http2Utilities.cs:line 982
   at Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.<>c__DisplayClass9_0.<<Reset_AfterResponseHeaders_Resets>b__1>d.MoveNext() in D:\github\AspNetCore\src\Servers\HttpSys\test\FunctionalTests\Http2Tests.cs:line 399
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Http2Cat.Http2CatHostedService.RunAsync() in D:\github\AspNetCore\src\Shared\Http2cat\Http2CatHostedService.cs:line 105
   at Microsoft.AspNetCore.Http2Cat.Http2CatHostedService.RunAsync() in D:\github\AspNetCore\src\Shared\Http2cat\Http2CatHostedService.cs:line 121
   at Microsoft.AspNetCore.Http2Cat.Http2CatHostedService.RunAsync() in D:\github\AspNetCore\src\Shared\Http2cat\Http2CatHostedService.cs:line 121
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
Result Message:	
System.AggregateException : One or more hosted services failed to stop. (Assert.Equal('RST_STREAM', 'DATA') failed)
---- System.Exception : Assert.Equal('RST_STREAM', 'DATA') failed

Test Name:	Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.Reset_CompleteAsyncDurringRequestBody_Resets
Test FullName:	Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.Reset_CompleteAsyncDurringRequestBody_Resets
Test Source:	D:\github\AspNetCore\src\Servers\HttpSys\test\FunctionalTests\Http2Tests.cs : line 605
Test Outcome:	Failed
Test Duration:	0:00:00

Test Name:	Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.Reset_CompleteAsyncDurringRequestBody_Resets
Test Outcome:	Failed
Result StackTrace:	
at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.WaitForShutdownAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.Reset_CompleteAsyncDurringRequestBody_Resets() in D:\github\AspNetCore\src\Servers\HttpSys\test\FunctionalTests\Http2Tests.cs:line 632
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
   at Microsoft.AspNetCore.Http2Cat.Http2Utilities.Assert.Equal[T](T expected, T actual) in D:\github\AspNetCore\src\Shared\Http2cat\Http2Utilities.cs:line 1059
   at Microsoft.AspNetCore.Http2Cat.Http2Utilities.VerifyDataFrame(Http2Frame frame, Int32 expectedStreamId, Boolean endOfStream, Int32 length) in D:\github\AspNetCore\src\Shared\Http2cat\Http2Utilities.cs:line 966
   at Microsoft.AspNetCore.Server.HttpSys.FunctionalTests.Http2Tests.<>c__DisplayClass14_0.<<Reset_CompleteAsyncDurringRequestBody_Resets>b__1>d.MoveNext() in D:\github\AspNetCore\src\Servers\HttpSys\test\FunctionalTests\Http2Tests.cs:line 651
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Http2Cat.Http2CatHostedService.RunAsync() in D:\github\AspNetCore\src\Shared\Http2cat\Http2CatHostedService.cs:line 105
   at Microsoft.AspNetCore.Http2Cat.Http2CatHostedService.RunAsync() in D:\github\AspNetCore\src\Shared\Http2cat\Http2CatHostedService.cs:line 121
   at Microsoft.AspNetCore.Http2Cat.Http2CatHostedService.RunAsync() in D:\github\AspNetCore\src\Shared\Http2cat\Http2CatHostedService.cs:line 121
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
Result Message:	
System.AggregateException : One or more hosted services failed to stop. (Assert.Equal('END_STREAM', 'NONE') failed)
---- System.Exception : Assert.Equal('END_STREAM', 'NONE') failed

[Tratcher]

All of the above tests are inspecting responses at the HTTP/2 frame level and each is receiving an unexpected empty data frame after the headers. It happens for any response that does not send data with the headers. This seems like a Windows issue, I'll ask them.

[Tratcher]

Confirmed, these tests caught a regression in Windows. This is a good example of why we want to test preview builds. This regression made it all the way to the public beta channel and we were the first to report it.