Settings_MaxConcurrentStreamsGet_Server test failure

[BrennanConroy]

System.Net.Http.HttpRequestException : The request was aborted.
---- System.Net.Http.Http2StreamException : The HTTP/2 server reset the stream. HTTP/2 error code 'REFUSED_STREAM' (0x7).
   at System.Net.Http.Http2Connection.ThrowRetry(String message, Exception innerException)
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
   at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders()
   at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, CancellationTokenSource cts, Boolean disposeCts, Int64 timeoutTime, CancellationToken originalCancellationToken)
   at Microsoft.AspNetCore.Testing.TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber) in /_/src/Testing/src/TaskExtensions.cs:line 31
   at Microsoft.AspNetCore.Testing.TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber) in /_/src/Testing/src/TaskExtensions.cs:line 31
   at Interop.FunctionalTests.HttpClientHttp2InteropTests.Settings_MaxConcurrentStreamsGet_Server(String scheme) in /_/src/Servers/Kestrel/test/Interop.FunctionalTests/HttpClientHttp2InteropTests.cs:line 1234
--- End of stack trace from previous location ---
----- Inner Stack Trace -----

[0.001s] [TestLifetime] [Information] Starting test Settings_MaxConcurrentStreamsGet_Server-https at 2020-10-11T10:03:17
[0.002s] [Microsoft.Extensions.Hosting.Internal.Host] [Debug] Hosting starting
[0.165s] [Microsoft.Hosting.Lifetime] [Information] Now listening on: https://127.0.0.1:37301
[0.166s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Debug] Loaded hosting startup assembly Interop.FunctionalTests
[0.166s] [Microsoft.Hosting.Lifetime] [Information] Application started. Press Ctrl+C to shut down.
[0.166s] [Microsoft.Hosting.Lifetime] [Information] Hosting environment: Production
[0.166s] [Microsoft.Hosting.Lifetime] [Information] Content root path: /datadisks/disk1/work/A9620907/w/B2D909A0/e/
[0.166s] [Microsoft.Extensions.Hosting.Internal.Host] [Debug] Hosting started
[0.167s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HM3DPJFMBLMP" accepted.
[0.169s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HM3DPJFMBLMP" started.
[0.230s] [Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware] [Debug] Connection "0HM3DPJFMBLMP" established using the following protocol: Tls12
[0.236s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending SETTINGS frame for stream ID 0 with length 18 and flags NONE
[0.242s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending WINDOW_UPDATE frame for stream ID 0 with length 4 and flags 0x0
[0.242s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received SETTINGS frame for stream ID 0 with length 6 and flags NONE
[0.242s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending SETTINGS frame for stream ID 0 with length 0 and flags ACK
[0.242s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received WINDOW_UPDATE frame for stream ID 0 with length 4 and flags 0x0
[0.243s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 1 with length 20 and flags END_STREAM, END_HEADERS
[0.243s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 3 with length 20 and flags END_STREAM, END_HEADERS
[0.243s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request starting HTTP/2 GET https://127.0.0.1:37301/ - -
[0.243s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request starting HTTP/2 GET https://127.0.0.1:37301/ - -
[0.243s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 5 with length 20 and flags END_STREAM, END_HEADERS
[0.243s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request starting HTTP/2 GET https://127.0.0.1:37301/ - -
[0.243s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 7 with length 20 and flags END_STREAM, END_HEADERS
[0.244s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request starting HTTP/2 GET https://127.0.0.1:37301/ - -
[0.244s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 9 with length 20 and flags END_STREAM, END_HEADERS
[0.244s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HM3DPJFMBLMP" reached the maximum number of concurrent HTTP/2 streams allowed.
[0.244s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 11 with length 20 and flags END_STREAM, END_HEADERS
[0.244s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request starting HTTP/2 GET https://127.0.0.1:37301/ - -
[0.245s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending HEADERS frame for stream ID 3 with length 45 and flags END_STREAM, END_HEADERS
[0.245s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HM3DPJFMBLMP": HTTP/2 stream error.
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2StreamErrorException: HTTP/2 stream ID 11 error (REFUSED_STREAM): A new stream was refused because this connection has reached its stream limit.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.StartStream() in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 1037
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.DecodeHeadersAsync(Boolean endHeaders, ReadOnlySequence`1& payload) in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 973
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessHeadersFrameAsync[TContext](IHttpApplication`1 application, ReadOnlySequence`1& payload) in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 618
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessFrameAsync[TContext](IHttpApplication`1 application, ReadOnlySequence`1& payload) in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 463
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application) in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 242
[0.246s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request finished HTTP/2 GET https://127.0.0.1:37301/ - - - 200 0 - 2.5278ms
[0.246s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending RST_STREAM frame for stream ID 11 with length 4 and flags 0x0
[0.246s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending HEADERS frame for stream ID 7 with length 7 and flags END_STREAM, END_HEADERS
[0.246s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 13 with length 20 and flags END_STREAM, END_HEADERS
[0.246s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request finished HTTP/2 GET https://127.0.0.1:37301/ - - - 200 0 - 2.4357ms
[0.246s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 15 with length 20 and flags END_STREAM, END_HEADERS
[0.246s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HM3DPJFMBLMP" reached the maximum number of concurrent HTTP/2 streams allowed.
[0.247s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request starting HTTP/2 GET https://127.0.0.1:37301/ - -
[0.247s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending HEADERS frame for stream ID 13 with length 7 and flags END_STREAM, END_HEADERS
[0.247s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 17 with length 20 and flags END_STREAM, END_HEADERS
[0.247s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request finished HTTP/2 GET https://127.0.0.1:37301/ - - - 200 0 - 0.4306ms
[0.247s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HM3DPJFMBLMP" reached the maximum number of concurrent HTTP/2 streams allowed.
[0.247s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 19 with length 20 and flags END_STREAM, END_HEADERS
[0.247s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request starting HTTP/2 GET https://127.0.0.1:37301/ - -
[0.247s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending HEADERS frame for stream ID 15 with length 7 and flags END_STREAM, END_HEADERS
[0.248s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request finished HTTP/2 GET https://127.0.0.1:37301/ - - - 200 0 - 0.3089ms
[0.248s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HM3DPJFMBLMP": HTTP/2 stream error.
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2StreamErrorException: HTTP/2 stream ID 19 error (REFUSED_STREAM): A new stream was refused because this connection has reached its stream limit.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.StartStream() in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 1037
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.DecodeHeadersAsync(Boolean endHeaders, ReadOnlySequence`1& payload) in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 973
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessHeadersFrameAsync[TContext](IHttpApplication`1 application, ReadOnlySequence`1& payload) in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 618
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessFrameAsync[TContext](IHttpApplication`1 application, ReadOnlySequence`1& payload) in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 463
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application) in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 242
[0.248s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending RST_STREAM frame for stream ID 19 with length 4 and flags 0x0
[0.248s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received SETTINGS frame for stream ID 0 with length 0 and flags ACK
[0.248s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request starting HTTP/2 GET https://127.0.0.1:37301/ - -
[0.249s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending HEADERS frame for stream ID 17 with length 7 and flags END_STREAM, END_HEADERS
[0.249s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request finished HTTP/2 GET https://127.0.0.1:37301/ - - - 200 0 - 1.5675ms
[0.250s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending HEADERS frame for stream ID 5 with length 7 and flags END_STREAM, END_HEADERS
[0.250s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request finished HTTP/2 GET https://127.0.0.1:37301/ - - - 200 0 - 6.7389ms
[0.251s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending HEADERS frame for stream ID 9 with length 7 and flags END_STREAM, END_HEADERS
[0.252s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" received HEADERS frame for stream ID 21 with length 20 and flags END_STREAM, END_HEADERS
[0.252s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request starting HTTP/2 GET https://127.0.0.1:37301/ - -
[0.253s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request finished HTTP/2 GET https://127.0.0.1:37301/ - - - 200 0 - 8.2426ms
[0.253s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending HEADERS frame for stream ID 21 with length 7 and flags END_STREAM, END_HEADERS
[0.253s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request finished HTTP/2 GET https://127.0.0.1:37301/ - - - 200 0 - 0.6910ms
[0.253s] [Microsoft.AspNetCore.Server.Kestrel] [Verbose] Connection id "0HM3DPJFMBLMP" sending HEADERS frame for stream ID 1 with length 7 and flags END_STREAM, END_HEADERS
[0.254s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request finished HTTP/2 GET https://127.0.0.1:37301/ - - - 200 0 - 10.6481ms
[0.271s] [Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets] [Debug] Connection id "0HM3DPJFMBLMP" received FIN.
[0.272s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HM3DPJFMBLMP" is closed. The last processed stream ID was 21.
[0.272s] [Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets] [Debug] Connection id "0HM3DPJFMBLMP" sending FIN because: "The client closed the connection."
[0.278s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HM3DPJFMBLMP" stopped.
[0.281s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Some connections failed to close gracefully during server shutdown.
[0.286s] [TestLifetime] [Information] Finished test Settings_MaxConcurrentStreamsGet_Server-https in 0.2845283s

Thanks for contacting us.
We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We will evaluate the request when we are planning the work for the next milestone. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[halter73]

One thing that might help make these HttpClientHttp2InteropTests more reliable (but probably won't fix this issue) is to use SocketHttpHandler's new ConnectCallback to wire it up with Kestrel's in-memory test transport like Http2EndToEndTests does.

aspnetcore/src/Servers/Kestrel/test/InMemory.FunctionalTests/Http2/Http2EndToEndTests.cs

Lines 47 to 61 in 540680a

	using var connection = server.CreateConnection();
	using var socketsHandler = new SocketsHttpHandler()
	{
	ConnectCallback = (_, _) =>
	{
	if (connectionCount != 0)
	{
	throw new InvalidOperationException();
	}
	connectionCount++;
	return new ValueTask<Stream>(connection.Stream);
	},
	};

If we do that, we should move it into InMemory.FunctionalTests.

[BrennanConroy]

We'll see what happens with dotnet/runtime#44669 before deciding what to do with this test.

[BrennanConroy]

Runtime fixed the issue, we should be able to unquarantine.