TcpReceiveSendGetsCanceledByDispose test fails on rhel7

[tmds]

These TcpReceiveSendGetsCanceledByDispose tests fail on our internal CI server on rhel7:

System.Net.Sockets.Tests.SendReceive_SpanSyncForceNonBlocking.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: True)
System.Net.Sockets.Tests.SendReceive_SpanSyncForceNonBlocking.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: True, dualModeClient: False)
System.Net.Sockets.Tests.SendReceive_Sync.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: True)
System.Net.Sockets.Tests.SendReceive_Sync.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: True, dualModeClient: False)
System.Net.Sockets.Tests.SendReceive_SyncForceNonBlocking.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: True)
System.Net.Sockets.Tests.SendReceive_SyncForceNonBlocking.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: True, dualModeClient: False)
System.Net.Sockets.Tests.SendReceive_SpanSync.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: True)
System.Net.Sockets.Tests.SendReceive_SpanSync.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: True, dualModeClient: False)

The tests that are failing are matching this condition:

runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs

Lines 1002 to 1005 in 5a47690

	// RHEL7 kernel has a bug preventing close(AF_UNKNOWN) to succeed with IPv6 sockets.
	// In this case Dispose will trigger a graceful shutdown, which means that receive will succeed on socket2.
	// TODO: Remove this, once CI machines are updated to a newer kernel.
	bool expectGracefulShutdown = UsesSync && PlatformDetection.IsRedHatFamily7 && receiveOrSend && (ipv6Server || dualModeClient);

The failure exception is masked by the Timeout ([Theory(Timeout = 40000)]) that is applied to this test:

        <failure exception-type="Xunit.Sdk.TestTimeoutException">
          <message><![CDATA[Test execution timed out after 40000 milliseconds]]></message>
          <stack-trace><![CDATA[]]></stack-trace>
        </failure>

cc @antonfirsov

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

These TcpReceiveSendGetsCanceledByDispose tests fail on our internal CI server on rhel7:

System.Net.Sockets.Tests.SendReceive_SpanSyncForceNonBlocking.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: True)
System.Net.Sockets.Tests.SendReceive_SpanSyncForceNonBlocking.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: True, dualModeClient: False)
System.Net.Sockets.Tests.SendReceive_Sync.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: True)
System.Net.Sockets.Tests.SendReceive_Sync.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: True, dualModeClient: False)
System.Net.Sockets.Tests.SendReceive_SyncForceNonBlocking.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: True)
System.Net.Sockets.Tests.SendReceive_SyncForceNonBlocking.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: True, dualModeClient: False)
System.Net.Sockets.Tests.SendReceive_SpanSync.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: False, dualModeClient: True)
System.Net.Sockets.Tests.SendReceive_SpanSync.TcpReceiveSendGetsCanceledByDispose(receiveOrSend: True, ipv6Server: True, dualModeClient: False)

The tests that are failing are matching this condition:

runtime/src/libraries/System.Net.Sockets/tests/FunctionalTests/SendReceive/SendReceive.cs

Lines 1002 to 1005 in 5a47690

	// RHEL7 kernel has a bug preventing close(AF_UNKNOWN) to succeed with IPv6 sockets.
	// In this case Dispose will trigger a graceful shutdown, which means that receive will succeed on socket2.
	// TODO: Remove this, once CI machines are updated to a newer kernel.
	bool expectGracefulShutdown = UsesSync && PlatformDetection.IsRedHatFamily7 && receiveOrSend && (ipv6Server || dualModeClient);

The failure exception is masked by the Timeout ([Theory(Timeout = 40000)]) that is applied to this test:

        <failure exception-type="Xunit.Sdk.TestTimeoutException">
          <message><![CDATA[Test execution timed out after 40000 milliseconds]]></message>
          <stack-trace><![CDATA[]]></stack-trace>
        </failure>

cc @antonfirsov

Author:	tmds
Assignees:	-
Labels:	area-System.Net.Sockets, untriaged
Milestone:	-

[tmds]

cc @omajid

[tmds]

I have debugged this further.

This looks like a regression in the RHEL kernel. I have installed RHEL 7.9 in a VM which came with kernel 3.10.0-1160.el7.x86_64. Tests passed.
I updated my system and kernel changed to 3.10.0-1160.25.1.el7.x86_64 and now tests fail.

I'll look at writing a reproducer in C and when it fails report it as a kernel bug.

When I got the reproducer, I'd like to make a PR to skip this combination. Otherwise our internal CI will fail until there is a patched kernel. The public CI will also fail when/if it moves to the latest RHEL7 kernel.

[tmds]

It turns out the test misbehaves on a system that has a bugfix for the kernel bug that is mentioned in the comments.
#52833 will fix this.