[PERF] Mono AOT BDN runs failing to build "Unable to compile method" #98967
Description
Description
Performance Mono AOT runs are failing to run due to failures occurring during BenchmarkDotNet. Potentially related to the performance BDN update causing: dotnet/performance#3984, based on initial testing but have some more looking to do.
FYI: @ilonatommy @matouskozak
Reproduction Steps
(Tested in WSL Ubuntu 22.4.3 LTS)
Clone performance repo
cd into performance/scripts
run (replace path): sudo python3 benchmarks_local.py --commits b29efed9995da044d97f9d3f97bc3d9de80a2050 --run-types MonoAOTLLVM --filter *Span.IndexerBench.CoveredIndex2* *WriteReadAsync*
Expected behavior
Test builds and runs successfully
Actual behavior
Multiple failures, full logs can be found here: https://dev.azure.com/dnceng/internal/_build/results?buildId=2387838&view=results, under the mono AOT micro_mono failing run. The seemingly top-level failure occuring during the BDN job build is:
[2024/02/26 12:00:36][INFO] [Microsoft.TestPlatform.CrossPlatEngine.dll] Exec (with response file contents expanded) in /home/helixbot/work/B8010A23/w/B11A09A0/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-HBDSBB/bin/net9.0/linux-x64/publish: MONO_PATH=/home/helixbot/work/B8010A23/w/B11A09A0/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-HBDSBB/bin/net9.0/linux-x64/publish: MONO_ENV_OPTIONS= /home/helixbot/work/B8010A23/p/monoaot/mono-aot-cross --debug --llvm "--aot=mcpu=native,nodebug,llvm-path=/home/helixbot/work/B8010A23/p/monoaot/pack/runtimes/linux-x64/native,outfile=/home/helixbot/work/B8010A23/w/B11A09A0/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-HBDSBB/bin/net9.0/linux-x64/publish/Microsoft.TestPlatform.CrossPlatEngine.dll.so,llvm-outfile=/home/helixbot/work/B8010A23/w/B11A09A0/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-HBDSBB/bin/net9.0/linux-x64/publish/Microsoft.TestPlatform.CrossPlatEngine.dll-llvm.o" "Microsoft.TestPlatform.CrossPlatEngine.dll"
[2024/02/26 12:00:36][INFO]
[2024/02/26 12:00:36][INFO] Unable to compile method 'Microsoft.VisualStudio.TestPlatform.ObjectModel.Engine.IProxyDiscoveryManager Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.TestEngine:GetDiscoveryManager (Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.IRequestData,Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.DiscoveryCriteria,System.Collections.Generic.IDictionary`2<string, Microsoft.VisualStudio.TestPlatform.ObjectModel.SourceDetail>,Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.IWarningLogger)' due to: 'VTable setup of type Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.InProcessProxyDiscoveryManager failed assembly:/home/helixbot/work/B8010A23/w/B11A09A0/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-HBDSBB/bin/net9.0/linux-x64/publish/Microsoft.TestPlatform.CrossPlatEngine.dll type:InProcessProxyDiscoveryManager member:(null)'.
[2024/02/26 12:00:36][INFO] Unable to compile method 'Microsoft.VisualStudio.TestPlatform.ObjectModel.Engine.IProxyExecutionManager Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.TestEngine:GetExecutionManager (Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.IRequestData,Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.TestRunCriteria,System.Collections.Generic.IDictionary`2<string, Microsoft.VisualStudio.TestPlatform.ObjectModel.SourceDetail>,Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.IWarningLogger)' due to: 'VTable setup of type Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.InProcessProxyExecutionManager failed assembly:/home/helixbot/work/B8010A23/w/B11A09A0/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-HBDSBB/bin/net9.0/linux-x64/publish/Microsoft.TestPlatform.CrossPlatEngine.dll type:InProcessProxyExecutionManager member:(null)'.
[2024/02/26 12:00:36][INFO] Unable to compile method 'Microsoft.VisualStudio.TestPlatform.ObjectModel.Engine.IProxyExecutionManager Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.TestEngine:CreateNonParallelExecutionManager (Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.IRequestData,Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.TestRunCriteria,bool,Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.TestRuntimeProviderInfo)' due to: 'VTable setup of type Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyExecutionManager failed assembly:/home/helixbot/work/B8010A23/w/B11A09A0/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-HBDSBB/bin/net9.0/linux-x64/publish/Microsoft.TestPlatform.CrossPlatEngine.dll type:ProxyExecutionManager member:(null)'.
[2024/02/26 12:00:36][INFO] =================================================================
[2024/02/26 12:00:36][INFO] Native Crash Reporting
[2024/02/26 12:00:36][INFO] =================================================================
[2024/02/26 12:00:36][INFO] Got a SIGSEGV while executing native code. This usually indicates
[2024/02/26 12:00:36][INFO] a fatal error in the mono runtime or one of the native libraries
[2024/02/26 12:00:36][INFO] used by your application.
[2024/02/26 12:00:36][INFO] =================================================================
[2024/02/26 12:00:36][INFO] =================================================================
[2024/02/26 12:00:36][INFO] Native stacktrace:
[2024/02/26 12:00:36][INFO] =================================================================
[2024/02/26 12:00:36][INFO] 0x562452742d81 - Unknown
[2024/02/26 12:00:36][INFO] 0x5624526e629e - Unknown
[2024/02/26 12:00:36][INFO] 0x5624526422a6 - Unknown
[2024/02/26 12:00:36][INFO] 0x7fcff8a42520 - Unknown
[2024/02/26 12:00:36][INFO] 0x5624526790bc - Unknown
[2024/02/26 12:00:36][INFO] 0x56245265338d - Unknown
[2024/02/26 12:00:36][INFO] 0x56245266da0e - Unknown
[2024/02/26 12:00:36][INFO] 0x56245264d904 - Unknown
[2024/02/26 12:00:36][INFO] 0x5624526c528c - Unknown
[2024/02/26 12:00:36][INFO] 0x5624526b58c4 - Unknown
[2024/02/26 12:00:36][INFO] 0x5624526b11ce - Unknown
[2024/02/26 12:00:36][INFO] 0x56245263d49b - Unknown
[2024/02/26 12:00:36][INFO] 0x5624526393e0 - Unknown
[2024/02/26 12:00:36][INFO] 0x5624524f996a - Unknown
[2024/02/26 12:00:36][INFO] 0x7fcff8a29d90 - Unknown
[2024/02/26 12:00:36][INFO] 0x7fcff8a29e40 - Unknown
[2024/02/26 12:00:36][INFO] 0x5624524f9839 - Unknown
[2024/02/26 12:00:36][INFO] =================================================================
[2024/02/26 12:00:36][INFO] External Debugger Dump:
[2024/02/26 12:00:36][INFO] =================================================================
[2024/02/26 12:00:36][INFO] [New LWP 552030]
[2024/02/26 12:00:36][INFO] [Thread debugging using libthread_db enabled]
[2024/02/26 12:00:36][INFO] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[2024/02/26 12:00:36][INFO] 0x00007fcff8aea42f in __GI___wait4 (pid=pid@entry=552031, stat_loc=stat_loc@entry=0x7ffc9c723cc0, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
[2024/02/26 12:00:36][INFO] 30 ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory.
[2024/02/26 12:00:36][INFO] Id Target Id Frame
[2024/02/26 12:00:36][INFO] * 1 Thread 0x7fcff8c3c480 (LWP 552029) "mono-aot-cross" 0x00007fcff8aea42f in __GI___wait4 (pid=pid@entry=552031, stat_loc=stat_loc@entry=0x7ffc9c723cc0, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
[2024/02/26 12:00:36][INFO] 2 Thread 0x7fcff7bff640 (LWP 552030) "SGen worker" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5624533b4128 <work_cond+40>) at ./nptl/futex-internal.c:57
[2024/02/26 12:00:36][INFO] Thread 2 (Thread 0x7fcff7bff640 (LWP 552030) "SGen worker"):
[2024/02/26 12:00:36][INFO] #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5624533b4128 <work_cond+40>) at ./nptl/futex-internal.c:57
[2024/02/26 12:00:36][INFO] #1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x5624533b4128 <work_cond+40>) at ./nptl/futex-internal.c:87
[2024/02/26 12:00:36][INFO] #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5624533b4128 <work_cond+40>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
[2024/02/26 12:00:36][INFO] #3 0x00007fcff8a93a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5624533b40d8 <lock>, cond=0x5624533b4100 <work_cond>) at ./nptl/pthread_cond_wait.c:503
[2024/02/26 12:00:36][INFO] #4 ___pthread_cond_wait (cond=0x5624533b4100 <work_cond>, mutex=0x5624533b40d8 <lock>) at ./nptl/pthread_cond_wait.c:627
[2024/02/26 12:00:36][INFO] #5 0x0000562452531fe3 in mono_os_cond_wait (cond=<optimized out>, mutex=<optimized out>) at /__w/1/s/src/mono/mono/sgen/../../mono/utils/mono-os-mutex.h:219
[2024/02/26 12:00:36][INFO] #6 get_work (worker_index=0, work_context=<optimized out>, do_idle=<optimized out>, job=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:164
[2024/02/26 12:00:36][INFO] #7 thread_func (data=0x0) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:195
[2024/02/26 12:00:36][INFO] #8 0x00007fcff8a94ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
[2024/02/26 12:00:36][INFO] #9 0x00007fcff8b26850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
[2024/02/26 12:00:36][INFO] Thread 1 (Thread 0x7fcff8c3c480 (LWP 552029) "mono-aot-cross"):
[2024/02/26 12:00:36][INFO] #0 0x00007fcff8aea42f in __GI___wait4 (pid=pid@entry=552031, stat_loc=stat_loc@entry=0x7ffc9c723cc0, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
[2024/02/26 12:00:36][INFO] #1 0x00007fcff8aea3ab in __GI___waitpid (pid=pid@entry=552031, stat_loc=stat_loc@entry=0x7ffc9c723cc0, options=options@entry=0) at ./posix/waitpid.c:38
[2024/02/26 12:00:36][INFO] #2 0x0000562452742eb2 in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843
[2024/02/26 12:00:36][INFO] #3 mono_dump_native_crash_info (signal=<optimized out>, mctx=mctx@entry=0x7ffc9c724830, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870
[2024/02/26 12:00:36][INFO] #4 0x00005624526e629e in mono_handle_native_crash (signal=0x5624521d5aeb "SIGSEGV", mctx=0x7ffc9c724830, info=0x7ffc9c724af0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2967
[2024/02/26 12:00:36][INFO] #5 0x00005624526422a6 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0x7ffc9c724af0, context=0x7ffc9c7249c0, debug_fault_addr=0x108) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3834
[2024/02/26 12:00:36][INFO] #6 <signal handler called>
[2024/02/26 12:00:36][INFO] #7 mono_method_to_ir (cfg=cfg@entry=0x5624552222d0, method=method@entry=0x5624543f4e40, start_bblock=start_bblock@entry=0x562455227260, end_bblock=end_bblock@entry=0x5624552273b0, return_var=0x562455227210, inline_args=<optimized out>, inline_offset=1, is_virtual_call=0) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:12283
[2024/02/26 12:00:36][INFO] #8 0x000056245265338d in inline_method (cfg=cfg@entry=0x5624552222d0, cmethod=0x5624543f4e40, fsig=<optimized out>, fsig@entry=0x5624550074d8, sp=<optimized out>, sp@entry=0x562455227108, ip=<optimized out>, ip@entry=0x7fcff83ca561 "(-\002", real_offset=<optimized out>, inline_always=<optimized out>, is_empty=<optimized out>) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:4852
[2024/02/26 12:00:36][INFO] #9 0x000056245266da0e in mono_method_to_ir (cfg=cfg@entry=0x5624552222d0, method=method@entry=0x5624543f4dc8, start_bblock=0x562455226ba8, start_bblock@entry=0x0, end_bblock=0x562455226cf8, end_bblock@entry=0x0, return_var=return_var@entry=0x0, inline_args=inline_args@entry=0x0, inline_offset=0, is_virtual_call=0) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:7984
[2024/02/26 12:00:36][INFO] #10 0x000056245264d904 in mini_method_compile (method=<optimized out>, method@entry=0x5624543f4dc8, opts=<optimized out>, flags=<optimized out>, flags@entry=(JIT_FLAG_AOT | JIT_FLAG_LLVM | JIT_FLAG_USE_CURRENT_CPU), parts=<optimized out>, parts@entry=0, aot_method_index=<optimized out>, aot_method_index@entry=553) at /__w/1/s/src/mono/mono/mini/mini.c:3506
[2024/02/26 12:00:36][INFO] #11 0x00005624526c528c in compile_method (acfg=acfg@entry=0x562454365810, method=0x5624543f4dc8) at /__w/1/s/src/mono/mono/mini/aot-compiler.c:9518
[2024/02/26 12:00:36][INFO] #12 0x00005624526b58c4 in compile_methods (acfg=0x562454365810) at /__w/1/s/src/mono/mono/mini/aot-compiler.c:13097
[2024/02/26 12:00:36][INFO] #13 aot_assembly (ass=<optimized out>, jit_opts=jit_opts@entry=374434303, aot_options=aot_options@entry=0x7ffc9c726498) at /__w/1/s/src/mono/mono/mini/aot-compiler.c:15183
[2024/02/26 12:00:36][INFO] #14 0x00005624526b11ce in mono_aot_assemblies (assemblies=assemblies@entry=0x5624543641a0, nassemblies=1, jit_opts=374434303, aot_options=<optimized out>) at /__w/1/s/src/mono/mono/mini/aot-compiler.c:15583
[2024/02/26 12:00:36][INFO] #15 0x000056245263d49b in main_thread_handler (user_data=user_data@entry=0x7ffc9c726720) at /__w/1/s/src/mono/mono/mini/driver.c:1424
[2024/02/26 12:00:36][INFO] #16 0x00005624526393e0 in mono_main (argc=<optimized out>, argv=<optimized out>) at /__w/1/s/src/mono/mono/mini/driver.c:2652
[2024/02/26 12:00:36][INFO] #17 0x00005624524f996a in mono_main_with_options (argc=1413434944, argv=0x8) at /__w/1/s/src/mono/mono/mini/main.c:36
[2024/02/26 12:00:36][INFO] #18 main (argc=<optimized out>, argv=<optimized out>) at /__w/1/s/src/mono/mono/mini/main.c:88
[2024/02/26 12:00:36][INFO] [Inferior 1 (process 552029) detached]
[2024/02/26 12:00:36][INFO] =================================================================
[2024/02/26 12:00:36][INFO] Basic Fault Address Reporting
[2024/02/26 12:00:36][INFO] =================================================================
[2024/02/26 12:00:36][INFO] Memory around native instruction pointer (0x5624526790bc):0x5624526790ac 4c 8b b5 d8 fd ff ff 4c 8b bd 50 ff ff ff 74 2b L......L..P...t+
[2024/02/26 12:00:36][INFO] 0x5624526790bc 89 90 08 01 00 00 48 8b 40 08 48 39 c8 75 f1 eb [email protected]..
[2024/02/26 12:00:36][INFO] 0x5624526790cc 1a 4c 89 e7 e8 fb d1 00 00 48 8b 7d 88 4c 8b b5 .L.......H.}.L..
[2024/02/26 12:00:36][INFO] 0x5624526790dc d8 fd ff ff 4c 8b bd 50 ff ff ff 45 85 ff 79 35 ....L..P...E..y5
[2024/02/26 12:00:36][INFO] /home/helixbot/work/B8010A23/w/B11A09A0/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-HBDSBB/BenchmarkDotNet.Autogenerated.csproj(75,4): error : Precompiling failed for /home/helixbot/work/B8010A23/w/B11A09A0/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-HBDSBB/bin/net9.0/linux-x64/publish/Microsoft.TestPlatform.CrossPlatEngine.dll
Regression?
No response
Known Workarounds
No response
Configuration
Dotnet runtime hash: b29efed
----------------------------------------------
Initializing logger 2024-02-26 11:59:47.389286
----------------------------------------------
Installing tools.
----------------------
Downloading DotNet Cli
----------------------
DotNet Install Path: '/home/helixbot/work/B8010A23/w/B11A09A0/e/performance/tools/dotnet/x64'
Downloading https://dot.net/v1/dotnet-install.sh
Downloaded https://dot.net/v1/dotnet-install.sh OK
$ pushd "/home/helixbot/work/B8010A23/w/B11A09A0/e/performance"
$ /home/helixbot/work/B8010A23/w/B11A09A0/e/performance/tools/dotnet/x64/dotnet-install.sh -InstallDir /home/helixbot/work/B8010A23/w/B11A09A0/e/performance/tools/dotnet/x64 -Architecture x64 -Version 9.0.100-preview.3.24126.3
dotnet-install: Attempting to download using primary link https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.3.24126.3/dotnet-sdk-9.0.100-preview.3.24126.3-linux-x64.tar.gz
curl: (22) The requested URL returned error: 404
dotnet-install: The resource at primary link 'https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.3.24126.3/dotnet-sdk-9.0.100-preview.3.24126.3-linux-x64.tar.gz' is not available.
dotnet-install: Attempting to download using primary link https://dotnetbuilds.azureedge.net/public/Sdk/9.0.100-preview.3.24126.3/dotnet-sdk-9.0.100-preview.3.24126.3-linux-x64.tar.gz
dotnet-install: Remote file https://dotnetbuilds.azureedge.net/public/Sdk/9.0.100-preview.3.24126.3/dotnet-sdk-9.0.100-preview.3.24126.3-linux-x64.tar.gz size is 242927673 bytes.
dotnet-install: Extracting zip from https://dotnetbuilds.azureedge.net/public/Sdk/9.0.100-preview.3.24126.3/dotnet-sdk-9.0.100-preview.3.24126.3-linux-x64.tar.gz
dotnet-install: Downloaded file size is 242927673 bytes.
dotnet-install: The remote and local file sizes are equal.
dotnet-install: Installed version is 9.0.100-preview.3.24126.3
dotnet-install: Adding to current process PATH: `/home/helixbot/work/B8010A23/w/B11A09A0/e/performance/tools/dotnet/x64`. Note: This change will be visible only when sourcing script.
dotnet-install: Note that the script does not resolve dependencies during installation.
dotnet-install: To check the list of dependencies, go to https://learn.microsoft.com/dotnet/core/install, select your operating system and check the "Dependencies" section.
dotnet-install: Installation finished successfully.
$ popd
$ dotnet --info
.NET SDK:
Version: 9.0.100-preview.3.24126.3
Commit: 94aea21ee6
Workload version: 9.0.100-manifests.06e7bcac
MSBuild version: 17.10.0-preview-24122-01+d4cb14fe4
Runtime Environment:
OS Name: ubuntu
OS Version: 22.04
OS Platform: Linux
RID: linux-x64
Base Path: /home/helixbot/work/B8010A23/w/B11A09A0/e/performance/tools/dotnet/x64/sdk/9.0.100-preview.3.24126.3/
.NET workloads installed:
There are no installed workloads to display.
Host:
Version: 9.0.0-preview.2.24123.1
Architecture: x64
Commit: 99b76018b6
.NET SDKs installed:
9.0.100-preview.3.24126.3 [/home/helixbot/work/B8010A23/w/B11A09A0/e/performance/tools/dotnet/x64/sdk]
.NET runtimes installed:
Microsoft.AspNetCore.App 9.0.0-preview.2.24121.1 [/home/helixbot/work/B8010A23/w/B11A09A0/e/performance/tools/dotnet/x64/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 9.0.0-preview.2.24123.1 [/home/helixbot/work/B8010A23/w/B11A09A0/e/performance/tools/dotnet/x64/shared/Microsoft.NETCore.App]
Other architectures found:
None
Environment variables:
DOTNET_ROOT [/home/helixbot/work/B8010A23/w/B11A09A0/e/performance/tools/dotnet/x64]
global.json file:
Not found
Learn more:
https://aka.ms/dotnet/info
Download .NET:
https://aka.ms/dotnet/download
Other information
No response