Skip to content

Regression in System.Tests.Perf_UInt64 (ToString) #79289

@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 41f57b7862dca221687a6cb638c35e33ee8e25db
Compare 09613f3ed6cb5ce62e955d2a1979115879d707bb
Diff Diff

Regressions in System.Collections.ContainsTrue<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Array - Duration of single invocation 269.70 μs 291.19 μs 1.08 0.15 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsTrue<String>.Array(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 291.1920761574074 > 283.4228556552553.
IsChangePoint: Marked as a change because one of 9/29/2022 7:42:19 AM, 10/11/2022 10:51:47 AM, 10/25/2022 2:34:53 PM, 12/1/2022 11:48:13 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -13.679745201453423 (T) = (0 -294301.5516878831) / Math.Sqrt((34689258.78466946 / (32)) + (26346676.045331515 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.08013880888349276 = (272466.41752655274 - 294301.5516878831) / 272466.41752655274 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 41f57b7862dca221687a6cb638c35e33ee8e25db
Compare 09613f3ed6cb5ce62e955d2a1979115879d707bb
Diff Diff

Regressions in System.Tests.Perf_UInt64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 1.20 ns 2.40 ns 2.01 0.21 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt64.ToString(value: 0)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.4033548563465605 > 1.4579393401132879.
IsChangePoint: Marked as a change because one of 10/3/2022 8:45:37 PM, 12/1/2022 11:48:13 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -20.882326878308504 (T) = (0 -2.6873065212221836) / Math.Sqrt((0.013914976613451827 / (32)) + (0.06680088451249501 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -1.0014129777233116 = (1.342704654728033 - 2.6873065212221836) / 1.342704654728033 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions