Same NuGet version used when benchmarking different packages #931
Description
As discussed in Gitter, my attempt to benchmark two different NuGet versions of FakeItEasy seem to be running the benchmarks against a single version. I'm using BenchmarkDotNet version 0.11.1.821 from the nightlies, and started from the example configuration added in #922.
I quote the Gitter conversation:
Hmmm. As I continue working with this, I think it's doing the wrong thing. I don't know if I'm doing something wrong or if there's a bug. I think we're always running the code from one NuGet version. I have a benchmark that's dramatically faster with my 4.10.0… version than the 4.9.1, but if I run the two jobs, I get numbers in line with 4.10 only:
BenchmarkDotNet=v0.11.1.821-nightly, OS=Windows 10.0.17134.345 (1803/April2018Update/Redstone4)
Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge), 1 CPU, 8 logical and 4 physical cores
Frequency=2143568 Hz, Resolution=466.5119 ns, Timer=TSC
.NET Core SDK=2.1.402
[Host] : .NET Core 2.1.4 (CoreCLR 4.6.26814.03, CoreFX 4.6.26814.02), 64bit RyuJIT
4.9.1 : .NET Core 2.1.4 (CoreCLR 4.6.26814.03, CoreFX 4.6.26814.02), 64bit RyuJIT
Job=4.9.1 NugetReferences=FakeItEasy 4.9.1 Toolchain=.NET Core 2.1
IterationCount=1 LaunchCount=1 WarmupCount=10
| Method | Mean | Error |
|---|---|---|
| Literal | 135.1 us | NA |
BenchmarkDotNet=v0.11.1.821-nightly, OS=Windows 10.0.17134.345 (1803/April2018Update/Redstone4)
Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge), 1 CPU, 8 logical and 4 physical cores
Frequency=2143568 Hz, Resolution=466.5119 ns, Timer=TSC
.NET Core SDK=2.1.402
[Host] : .NET Core 2.1.4 (CoreCLR 4.6.26814.03, CoreFX 4.6.26814.02), 64bit RyuJIT
4.10.0-make-servicelocat001 : .NET Core 2.1.4 (CoreCLR 4.6.26814.03, CoreFX 4.6.26814.02), 64bit RyuJIT
Job=4.10.0-make-servicelocat001 NugetReferences=FakeItEasy 4.10.0-make-servicelocat001 Toolchain=.NET Core 2.1
IterationCount=1 LaunchCount=1 WarmupCount=10
| Method | Mean | Error |
|---|---|---|
| Literal | 15.23 us | NA |
BenchmarkDotNet=v0.11.1.821-nightly, OS=Windows 10.0.17134.345 (1803/April2018Update/Redstone4)
Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge), 1 CPU, 8 logical and 4 physical cores
Frequency=2143568 Hz, Resolution=466.5119 ns, Timer=TSC
.NET Core SDK=2.1.402
[Host] : .NET Core 2.1.4 (CoreCLR 4.6.26814.03, CoreFX 4.6.26814.02), 64bit RyuJIT
4.10.0-make-servicelocat001 : .NET Core 2.1.4 (CoreCLR 4.6.26814.03, CoreFX 4.6.26814.02), 64bit RyuJIT
4.9.1 : .NET Core 2.1.4 (CoreCLR 4.6.26814.03, CoreFX 4.6.26814.02), 64bit RyuJIT
Toolchain=.NET Core 2.1 IterationCount=1 LaunchCount=1
WarmupCount=10
| Method | Job | NugetReferences | Mean | Error |
|---|---|---|---|---|
| Literal | 4.10.0-make-servicelocat001 | FakeItEasy 4.10.0-make-servicelocat001 | 15.24 us | NA |
| Literal | 4.9.1 | FakeItEasy 4.9.1 | 15.31 us | NA |
My benchmark project references 4.9.1, and build my configuration looks like this:
public Config(params string[] versions)
{
// Specify jobs with different versions of the same Nuget package to benchmark.
// The Nuget versions referenced on these jobs must be greater or equal to the
// same Nuget version referenced in this benchmark project.
foreach (var version in versions)
{
Add(Job.MediumRun
.WithIterationCount(1)
.WithLaunchCount(1)
.With(CsProjCoreToolchain.Current.Value)
.WithNuget("FakeItEasy", version)
.WithId(version));
}
Add(DefaultConfig.Instance.GetColumnProviders().ToArray());
Add(DefaultConfig.Instance.GetLoggers().ToArray());
Add(CsvExporter.Default);
Add(MarkdownExporter.GitHub);
}the 4.10.0-make-servicelocat001 package is not an official one, but existed in a local NuGet source I had on my computer.
FakeItEasy.Benchmark.zip contains the source I used to reproduce, including the output I got when I ran benchmark.ps1, which attempts to benchmark 4.9.1 alone, 4.10.0-make-servicelocat001 alone, and the two together. The last attempt gives the suspect results.
I included nuget/4.10.0-make-servicelocat001.nupkg in case someone wants to reproduce.
In the .log file for the combined run, I see start dotnet add package FakeItEasy -v 4.10.0-make-servicelocat001 only. No similar line for 4.9.1, even when I change my .csproj to reference something older, like 4.8.0.
I'll try to debug myself, but am occupied for the rest of today. With any luck, I'm doing something silly.