Flaky test: WritingToConnectionAfterUnobservedCloseTriggersRequestAbortedToken

[ryanbrandenburg]

This test fails occasionally with the following error:

Moq.MockException : 
Expected invocation on the mock once, but was 0 times: t => t.ConnectionStop(It.IsAny<String>())
No setups configured.
Performed invocations:
KestrelTrace.IsEnabled(Critical)
KestrelTrace.BeginScope<ConnectionLogScope>([[ConnectionId, 0HLFHOOITD5M5]])
KestrelTrace.ConnectionStart("0HLFHOOITD5M5")
KestrelTrace.ConnectionDisconnect("0HLFHOOITD5M5")
KestrelTrace.ApplicationError("0HLFHOOITD5M5", "0HLFHOOITD5M5:00000001", System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.ValidateState(CancellationToken cancellationToken) in /_/src/Kestrel.Core/Internal/Http/HttpResponseStream.cs:line 168
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken) in /_/src/Kestrel.Core/Internal/Http/HttpResponseStream.cs:line 111
   at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ResponseTests.<>c__DisplayClass61_0.<<WritingToConnectionAfterUnobservedCloseTriggersRequestAbortedToken>b__6>d.MoveNext() in /_/test/Kestrel.FunctionalTests/ResponseTests.cs:line 2398
--- End of stack trace from previous location where exception was thrown ---
   at Xunit.Assert.RecordExceptionAsync(Func`1 testCode) in C:\Dev\xunit\xunit\src\xunit.assert\Asserts\Record.cs:line 82)
   at Moq.Mock.ThrowVerifyException(MethodCall expected, IEnumerable`1 setups, IEnumerable`1 actualCalls, Expression expression, Times times, Int32 callCount)
   at Moq.Mock.VerifyCalls(Interceptor targetInterceptor, MethodCall expected, Expression expression, Times times)
   at Moq.Mock.Verify[T](Mock`1 mock, Expression`1 expression, Times times, String failMessage)
   at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.ResponseTests.WritingToConnectionAfterUnobservedCloseTriggersRequestAbortedToken(ListenOptions listenOptions) in /_/test/Kestrel.FunctionalTests/ResponseTests.cs:line 2432
--- End of stack trace from previous location where exception was thrown ---

[ryanbrandenburg]

ServerCanAbortConnectionAfterUnobservedClose failed with about the same error.

This comment was made automatically. If there is a problem contact @ryanbrandenburg.

[ryanbrandenburg]

@muratg was this resolved? Please assign someone to find out.

[JunTaoLuo]

Looks like this is going to require back porting #2646 back to 2.1. The test failures are only showing up in 2.1 builds which lack this change and 2.2/master have been consistently passing after this change went it. This is a product change so we will need to ask ship-room. cc @muratg

[muratg]

@Eilon should we prep this for 2.1.5? It'll be a product change on connection lifetime improvements.

[Eilon]

We should spend some time to figure out if there's a way to make the test more reliable without changing the product code (i.e. anything in src/). This would be a very tough sell to include in LTS unless customers have reported this issue and are blocked.

[JunTaoLuo]

@halter73 Would it be better to assert ConnectionStop is called at most once instead of exactly once?

[JunTaoLuo]

Test fix in 2.1 is in. No src change made.