System.Net.* and System.Security.* crash on NativeAOT arm64

[karelz]

Frequency as of:

• 9/9 - last 30 days of System.Net.Security.Tests (with Core Dump) - all arm64-NativeAOT only
• 9/9 - last 30 days of System.Net.WebSockets.Client.Tests (with Core Dump) - all arm64-NativeAOT only
• 9/9 - last 30 days of System.Net.Http.Functional.Tests (with Core Dump) - mostly arm64-NativeAOT with 2 exceptions crossed out in the table
• 9/9 - last 30 days of System.Security.Cryptography.Tests (with Core Dump) - all arm64-NativeAOT only

Day	Test suite	Run	OS	Details
9/8	System.Security.Cryptography.Tests	PR 9648 - PR #75306 (main)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Debian.11.Arm64.Open)Ubuntu.1804.Armarch.Open	Core Dump - SIGILL
9/8	System.Security.Cryptography.Tests	PR 9492 - PR #75213 (main)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump - SIGILL
9/4	System.Net.WebSockets.Client.Tests	Rolling run 4489 (main)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Debian.11.Arm64.Open)Ubuntu.1804.Armarch.Open	Core Dump - SIGILL
9/2	System.Net.Http.Functional.Tests	Rolling run 3461 (main)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump - SIGILL
9/1	System.Net.Http.Functional.Tests	Rolling run 1981002 (7.0-rc1)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump - SIGILL
8/28	System.Net.Security.Tests	Rolling run 1972481 (main)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Debian.11.Arm64.Open)Ubuntu.1804.Armarch.Open	Core Dump - SIGILL
8/28	System.Net.Http.Functional.Tests	Rolling run 1972750	net7.0-windows-Release-x64-NativeAOT_Release-Windows.81.Amd64.Open	Core Dump - Unhandled Exception: System.NullReferenceException
8/28	System.Net.Http.Functional.Tests	Rolling run 1972413 (7.0)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump - SIGILL
8/27	System.Net.WebSockets.Client.Tests	Rolling run 1972127 (main)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Debian.11.Arm64.Open)Ubuntu.1804.Armarch.Open	Core Dump - SIGILL
8/26	System.Net.Security.Tests	Rolling run 1969936 (7.0)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump - SIGILL
8/25	System.Net.Http.Functional.Tests	Merged PR 1966082	net7.0-windows-Release-x64-NativeAOT_Release-Windows.81.Amd64.Open	Core Dump - [FAIL] System.Net.Http.Functional.Tests.SyncHttpHandlerTest_AutoRedirect.AllowAutoRedirect_True_ValidateNewMethodUsedOnRedirection
8/21	System.Net.Http.Functional.Tests	Rolling run 1957446 (7.0-rc1)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump - SIGILL
8/20	System.Net.Security.Tests	Rolling run 1956604 (main)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump - SIGILL
8/20	System.Net.WebSockets.Client.Tests	Rolling run 1955765 (7.0-rc1)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump - SIGILL

Not quite blocking CI yet, but close ...

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Frequency as of 8/27 - last 30 days (with Core dump):

Day	Run	OS	Details
8/26	Rolling run 1969936 (7.0)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump
8/20	Rolling run 1956604 (main)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump

Not quite blocking CI yet, but close ...

Author:	karelz
Assignees:	-
Labels:	arch-arm64, area-System.Net.Security
Milestone:	-

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Frequency as of 8/27 - last 30 days (with Core dump):

Day	Run	OS	Details
8/26	Rolling run 1969936 (7.0)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump
8/20	Rolling run 1956604 (main)	net7.0-Linux-Release-arm64-NativeAOT_Release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open	Core Dump

Not quite blocking CI yet, but close ...

Author:	karelz
Assignees:	-
Labels:	arch-arm64, area-System.Net.Security
Milestone:	-

[jkotas]

Stacktrace:

0x0000007f`84350000
libcrypto_so_1!X509_get_pubkey_parameters+0xf18
libcrypto_so_1!X509_verify_cert+0x78
System_Net_Security!System_Security_Cryptography_Interop_Crypto___CryptoNative_X509VerifyCert_g____PInvoke_275_0+0x3c [/_/src/libraries/System.Security.Cryptography/src/Microsoft.Interop.LibraryImportGenerator/Microsoft.Interop.LibraryImportGenerator/LibraryImports.g.cs @ 7264] 
System_Net_Security!System_Security_Cryptography_Interop_Crypto::CryptoNative_X509VerifyCert+0x38 [/_/src/libraries/System.Security.Cryptography/src/Microsoft.Interop.LibraryImportGenerator/Microsoft.Interop.LibraryImportGenerator/LibraryImports.g.cs @ 5839] 
System_Net_Security!System_Security_Cryptography_Interop_Crypto::X509VerifyCert+0x1c [/_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.X509.cs @ 196] 
System_Net_Security!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_OpenSslX509ChainProcessor::BuildWorkingChain+0x98 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/OpenSslX509ChainProcessor.cs @ 653] 
System_Net_Security!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_OpenSslX509ChainProcessor::Finish+0x58 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/OpenSslX509ChainProcessor.cs @ 688] 
System_Net_Security!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_ChainPal::BuildChainCore+0x320 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/ChainPal.OpenSsl.cs @ 198] 
System_Net_Security!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_ChainPal::BuildChain+0xb4 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/ChainPal.OpenSsl.cs @ 49] 
System_Net_Security!System_Security_Cryptography_System_Security_Cryptography_X509Certificates_X509Chain::Build+0x308 [/_/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/X509Chain.cs @ 126] 
System_Net_Security!System_Net_Security_System_Net_Security_SslStreamCertificateContext::Create+0x184 [/_/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamCertificateContext.cs @ 56] 
System_Net_Security!System_Net_Security_System_Net_Security_SslAuthenticationOptions::UpdateOptions+0x174 [/_/src/libraries/System.Net.Security/src/System/Net/Security/SslAuthenticationOptions.cs @ 123] 
System_Net_Security!System_Net_Security_System_Net_Security_SslStream::AuthenticateAsServerAsync+0x24 [/_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs @ 426] 
System_Net_Security!System_Net_Security_System_Net_Security_SslStream::AuthenticateAsServerAsync+0x8c [/_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs @ 419] 
System_Net_Security!System_Net_Security_Tests_System_Net_Security_Tests_AsyncSslStreamSystemDefaultTest::AuthenticateServerAsync+0x48 [/_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSystemDefaultsTest.cs @ 231] 
System_Net_Security!System_Net_Security_Tests_System_Net_Security_Tests_SslStreamSystemDefaultTest___c__DisplayClass9_0::<ClientAndServer_OneUsesDefault_OtherUsesLowerProtocol_Fails>b__0+0x70 [/_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSystemDefaultsTest.cs @ 125] 
System_Net_Security!xunit_assert_Xunit_Assert__RecordExceptionAsync_d__95::MoveNext+0x38 [/_/src/xunit.assert/Asserts/Record.cs @ 104] 
System_Net_Security!S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore::Start+0x40 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 43] 
System_Net_Security!xunit_assert_Xunit_Assert__RecordExceptionAsync+0x30 [/_/src/xunit.assert/Asserts/Record.cs @ 75] 
System_Net_Security!xunit_assert_Xunit_Assert__ThrowsAnyAsync_d__67_1<System___Canon>::MoveNext+0x48 [/_/src/xunit.assert/Asserts/ExceptionAsserts.cs @ 132] 
System_Net_Security!S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore::Start+0x64 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 43] 
System_Net_Security!xunit_assert_Xunit_Assert__ThrowsAnyAsync<System___Canon>+0x44 [/_/src/xunit.assert/Asserts/ExceptionAsserts.cs @ 119] 
System_Net_Security!System_Net_Security_Tests_System_Net_Security_Tests_SslStreamSystemDefaultTest__ClientAndServer_OneUsesDefault_OtherUsesLowerProtocol_Fails_d__9::MoveNext+0x1ac [/_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSystemDefaultsTest.cs @ 125] 
System_Net_Security!S_P_CoreLib_System_Runtime_CompilerServices_AsyncMethodBuilderCore::Start+0x40 [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 43] 
System_Net_Security!System_Net_Security_Tests_System_Net_Security_Tests_SslStreamSystemDefaultTest__ClientAndServer_OneUsesDefault_OtherUsesLowerProtocol_Fails+0x38 [/_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSystemDefaultsTest.cs @ 30] 

[jkotas]

A function pointer in libcrypto data structures is bad. Use-after-free bug?

84350000 is the corrupted function pointer. 8490... are the good function pointers around it.

0000007f`7c0dbaf0  0000007f`8490f1b0 0000007f`84350000
0000007f`7c0dbb00  0000007f`8490b8e0 0000007f`8490e290

[karelz]

Triage: We should treat it as high pri for 7.0 until we have further details about root cause.

[karelz]

cc @LakshanF as it seems to be specific to arm64 NativeAOT -- I think you mentioned there were problems with arm64 NativeAOT recently ...

[jkotas]

I have taken a look at the two most recent core dumps. It is still same pattern - function pointer overwritten in libcrypto
data structure.

My guess is that this crash is another manifestation of the problem that is behind other mysterious crashes in networking tests: #74795 and #72830 . There seems to be a stray memory write in networking that corrupts random memory.

[MichalStrehovsky]

I was looking at the System.Security.Cryptography.Tests crash in #75306 that also looks similar (we end up in some location that is not valid code) - https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-75306-merge-171bd00681b64e6184/System.Security.Cryptography.Tests/1/console.108589f8.log?helixlogtype=result.

That one is not a networking test, but libcrypto is the common theme. Maybe a crypto bug instead of networking bug?

cc @LakshanF as it seems to be specific to arm64 NativeAOT -- I think you mentioned there were problems with arm64 NativeAOT recently ...

I'm not aware of any active ARM64 issues with NativeAOT. The one we had was fixed two weeks ago.