Quarantine 2 flaky tests

[wtgodbe]

These tests failed in the following 2 builds:

Settings_MaxConcurrentStreamsPost_Server

System.Net.Http.HttpRequestException : The request was aborted.

---- System.Net.Http.Http2StreamException : The HTTP/2 server reset the stream. HTTP/2 error code 'REFUSED_STREAM' (0x7).

at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders()
at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts, CancellationToken callerToken, Int64 timeoutTime) at Microsoft.AspNetCore.Testing.TaskExtensions.TimeoutAfter[T](Task1 task, TimeSpan timeout, String filePath, Int32 lineNumber) in //src/Testing/src/TaskExtensions.cs:line 29
at Microsoft.AspNetCore.Testing.TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber) in //src/Testing/src/TaskExtensions.cs:line 29
at Interop.FunctionalTests.HttpClientHttp2InteropTests.Settings_MaxConcurrentStreamsPost_Server(String scheme) in /_/src/Servers/Kestrel/test/Interop.FunctionalTests/HttpClientHttp2InteropTests.cs:line 1300
--- End of stack trace from previous location ---
----- Inner Stack Trace -----

DoesNotThrowObjectDisposedExceptionFromWriteAsyncAfterConnectionIsAborted

System.IO.IOException : Received an unexpected EOF or 0 bytes from the transport stream.

at System.Net.Security.SslStream.g__InternalFillHandshakeBufferAsync|195_0[TIOAdapter](TIOAdapter adap, ValueTask`1 task, Int32 minSize)
at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter)
at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
at Microsoft.AspNetCore.Server.Kestrel.InMemory.FunctionalTests.HttpsTests.DoesNotThrowObjectDisposedExceptionFromWriteAsyncAfterConnectionIsAborted() in //src/Servers/Kestrel/test/InMemory.FunctionalTests/HttpsTests.cs:line 250
at Microsoft.AspNetCore.Server.Kestrel.InMemory.FunctionalTests.HttpsTests.DoesNotThrowObjectDisposedExceptionFromWriteAsyncAfterConnectionIsAborted() in //src/Servers/Kestrel/test/InMemory.FunctionalTests/HttpsTests.cs:line 260
--- End of stack trace from previous location ---

[halter73]

It looks like DoesNotThrowObjectDisposedExceptionFromWriteAsyncAfterConnectionIsAborted test is failing due to hitting Kestrel's default 10 second handshake timeout. We could increase this, but I feel that shouldn't be necessary. Hopefully it doesn't fail again in the next month and leaves quarantine.

[0.001s] [TestLifetime] [Information] Starting test DoesNotThrowObjectDisposedExceptionFromWriteAsyncAfterConnectionIsAborted at 2020-03-19T23:57:37
[0.006s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Debug] Hosting starting
[0.018s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Debug] Hosting started
[0.018s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Debug] Loaded hosting startup assembly testhost
[1.247s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HLUCC522N5MD" accepted.
[2.931s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HLUCC522N5MD" started.
[15.902s] [Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware] [Debug] Authentication of the HTTPS connection timed out.
[15.903s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HLUCC522N5MD" stopped.
[15.917s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Debug] Hosting shutdown
[16.162s] [TestLifetime] [Information] Finished test DoesNotThrowObjectDisposedExceptionFromWriteAsyncAfterConnectionIsAborted in 16.1608058s

In the Settings_MaxConcurrentStreamsPost_Server test, we expect Kestrel to respond with a "'REFUSED_STREAM' (0x7)" RST_STREAM frame. The expectation is that HttpClient retries the request. Here's the rather long and hard to find server side logs for that.

Comparing the failed log to a local passed test, I see 5 refused streams in the failed case instead of 1 refused stream when I test locally. I don't think this really explains why the test is failing though. It looks like HttpClient only retries 4 of the 5 refused requests, but I don't know why.

When I ran the test locally, I also noticed the non-https version failed once with "ENHANCE_YOUR_CALM" in my logs. I would be suspicious that's related if it wasn't for the CI failure without that in the logs. You have any ideas @Tratcher?