Skip to content

[Perf] Linux/arm64: 93 Improvements on 4/4/2023 3:37:20 PM #15800

@performanceautofiler

Description

@performanceautofiler

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AndNotBenchmark - Duration of single invocation 17.63 ns 1.50 ns 0.09 0.14 False
ConditionalSelectBenchmark - Duration of single invocation 33.08 ns 1.44 ns 0.04 0.34 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Int32>.AndNotBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.5020280866057338 < 16.876489203384804.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 2:19:19 PM falls between 4/2/2023 6:09:37 PM and 4/11/2023 2:19:19 PM.
IsImprovementStdDev: Marked as improvement because 45.29641338397006 (T) = (0 -1.276750113372158) / Math.Sqrt((51.171374208213244 / (299)) + (0.014183315757627846 / (24))) is greater than 1.967381707010986 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (24) - 2, .975) and 0.9363149410344181 = (20.047875186268897 - 1.276750113372158) / 20.047875186268897 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int32>.ConditionalSelectBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.443070698756123 < 31.08837401644105.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 2:19:19 PM falls between 4/2/2023 6:09:37 PM and 4/11/2023 2:19:19 PM.
IsImprovementStdDev: Marked as improvement because 40.92405769771871 (T) = (0 -1.3872979912418957) / Math.Sqrt((366.5470219657404 / (299)) + (0.3271617419731319 / (24))) is greater than 1.967381707010986 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (24) - 2, .975) and 0.9704515708018088 = (46.94997429260359 - 1.3872979912418957) / 46.94997429260359 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XorBenchmark - Duration of single invocation 13.26 ns 2.72 ns 0.21 0.14 False
OnesComplementOperatorBenchmark - Duration of single invocation 11.01 ns 1.25 ns 0.11 0.17 False
SubtractBenchmark - Duration of single invocation 50.92 ns 1.24 ns 0.02 0.11 False
SubtractionOperatorBenchmark - Duration of single invocation 52.54 ns 1.24 ns 0.02 0.14 False
MultiplyOperatorBenchmark - Duration of single invocation 54.06 ns 1.56 ns 0.03 0.06 True
UnaryNegateOperatorBenchmark - Duration of single invocation 48.20 ns 1.31 ns 0.03 0.11 False
NegateBenchmark - Duration of single invocation 49.05 ns 0.94 ns 0.02 0.11 False
AddBenchmark - Duration of single invocation 51.73 ns 1.27 ns 0.02 0.10 False
OnesComplementBenchmark - Duration of single invocation 11.04 ns 1.20 ns 0.11 0.16 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.XorBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 2.7245058861569293 < 12.65535593637901.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 2:19:19 PM falls between 4/2/2023 6:09:37 PM and 4/11/2023 2:19:19 PM.
IsImprovementStdDev: Marked as improvement because 45.634072534363995 (T) = (0 -1.3017696084431034) / Math.Sqrt((19.24830514105451 / (299)) + (0.16349192183846423 / (24))) is greater than 1.967381707010986 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (24) - 2, .975) and 0.9034110155736593 = (13.477412731633393 - 1.3017696084431034) / 13.477412731633393 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.OnesComplementOperatorBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2482335076546036 < 10.741597713691915.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 2:19:19 PM falls between 4/2/2023 6:09:37 PM and 4/11/2023 2:19:19 PM.
IsImprovementStdDev: Marked as improvement because 46.4785397896914 (T) = (0 -1.179297446179264) / Math.Sqrt((7.2959683174300265 / (299)) + (0.014112032277604719 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8617531462587397 = (8.530374574646096 - 1.179297446179264) / 8.530374574646096 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.SubtractBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.243234803944142 < 49.33380391953601.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 2:19:19 PM falls between 4/2/2023 6:09:37 PM and 4/11/2023 2:19:19 PM.
IsImprovementStdDev: Marked as improvement because 49.41086665179494 (T) = (0 -1.2079357588081303) / Math.Sqrt((274.4877779420455 / (299)) + (0.012054498776054011 / (24))) is greater than 1.967381707010986 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (24) - 2, .975) and 0.9751264586754723 = (48.56307925952596 - 1.2079357588081303) / 48.56307925952596 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.SubtractionOperatorBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2397742455154066 < 49.584629086518355.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 2:19:19 PM falls between 4/2/2023 6:09:37 PM and 4/11/2023 2:19:19 PM.
IsImprovementStdDev: Marked as improvement because 49.41664081373533 (T) = (0 -1.1915122024710068) / Math.Sqrt((272.2498188882269 / (299)) + (0.033418113959651066 / (24))) is greater than 1.967381707010986 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (24) - 2, .975) and 0.9753727526477725 = (48.38186685785807 - 1.1915122024710068) / 48.38186685785807 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.MultiplyOperatorBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.5648411396123767 < 51.32172022899524.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 2:19:19 PM falls between 4/2/2023 6:09:37 PM and 4/11/2023 2:19:19 PM.
IsImprovementStdDev: Marked as improvement because 420.70708586717114 (T) = (0 -1.1855661356865292) / Math.Sqrt((4.354876964850811 / (299)) + (0.043330868416910646 / (24))) is greater than 1.967381707010986 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (24) - 2, .975) and 0.9784495363315451 = (55.01348620270908 - 1.1855661356865292) / 55.01348620270908 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.UnaryNegateOperatorBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.3072416693473092 < 46.73957786809049.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 2:19:19 PM falls between 4/2/2023 6:09:37 PM and 4/11/2023 2:19:19 PM.
IsImprovementStdDev: Marked as improvement because 49.507550671468024 (T) = (0 -1.16834132302815) / Math.Sqrt((227.9045536045062 / (299)) + (0.013693768550479964 / (24))) is greater than 1.967381707010986 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (24) - 2, .975) and 0.973690308901839 = (44.40726113693591 - 1.16834132302815) / 44.40726113693591 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.NegateBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.939447504593018 < 46.78929522642719.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 2:19:19 PM falls between 4/2/2023 6:09:37 PM and 4/11/2023 2:19:19 PM.
IsImprovementStdDev: Marked as improvement because 49.371969759928845 (T) = (0 -1.2396301329822899) / Math.Sqrt((228.04450812563186 / (299)) + (0.04835197353727195 / (24))) is greater than 1.967381707010986 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (24) - 2, .975) and 0.9720892971149595 = (44.414149585846 - 1.2396301329822899) / 44.414149585846 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.AddBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2727066191748166 < 49.445510250254735.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 2:19:19 PM falls between 4/2/2023 6:09:37 PM and 4/11/2023 2:19:19 PM.
IsImprovementStdDev: Marked as improvement because 49.427645253215395 (T) = (0 -1.2112582243740047) / Math.Sqrt((274.07654471451104 / (299)) + (0.028619866844746288 / (24))) is greater than 1.967381707010986 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (24) - 2, .975) and 0.9750589328721894 = (48.56481152818788 - 1.2112582243740047) / 48.56481152818788 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.OnesComplementBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1951175201577973 < 10.729606784943838.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 2:19:19 PM falls between 4/2/2023 6:09:37 PM and 4/11/2023 2:19:19 PM.
IsImprovementStdDev: Marked as improvement because 46.343767946393456 (T) = (0 -1.2054481705421518) / Math.Sqrt((7.286629522207985 / (299)) + (0.01509326054979665 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8587952952507809 = (8.536883899747105 - 1.2054481705421518) / 8.536883899747105 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<UInt64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseOrOperatorBenchmark - Duration of single invocation 9.56 ns 1.37 ns 0.14 0.15 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 9.61 ns 1.34 ns 0.14 0.22 False
BitwiseAndOperatorBenchmark - Duration of single invocation 9.30 ns 1.23 ns 0.13 0.19 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt64>.BitwiseOrOperatorBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.3708428966244965 < 9.062077201378779.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 2:19:19 PM falls between 4/2/2023 6:09:37 PM and 4/11/2023 2:19:19 PM.
IsImprovementStdDev: Marked as improvement because 41.80389853122208 (T) = (0 -1.3010475364995557) / Math.Sqrt((8.241322850189723 / (299)) + (0.19633688144707123 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8592505866804663 = (9.243715521185702 - 1.3010475364995557) / 9.243715521185702 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt64>.ExclusiveOrOperatorBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.3441396750716972 < 9.073448341276944.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 2:19:19 PM falls between 4/2/2023 6:09:37 PM and 4/11/2023 2:19:19 PM.
IsImprovementStdDev: Marked as improvement because 45.54850972272243 (T) = (0 -1.2711240227582106) / Math.Sqrt((8.317176562976902 / (299)) + (0.06289602162541134 / (24))) is greater than 1.967381707010986 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (24) - 2, .975) and 0.8620989082811978 = (9.217650178942705 - 1.2711240227582106) / 9.217650178942705 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt64>.BitwiseAndOperatorBenchmark


Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2335006804555422 < 9.11753036150609.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 2:19:19 PM falls between 4/2/2023 6:09:37 PM and 4/11/2023 2:19:19 PM.
IsImprovementStdDev: Marked as improvement because 45.82081553103783 (T) = (0 -1.2090388615557626) / Math.Sqrt((8.25185632605875 / (299)) + (0.07121846498242342 / (24))) is greater than 1.967381707010986 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (24) - 2, .975) and 0.8688664325282894 = (9.219903681920265 - 1.2090388615557626) / 9.219903681920265 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions