Using WithCustomBuildConfiguration leads to always running with RyuJIT Debug

[abelbraaksma]

Recently I got some help from you guys to set up a way to run benchmarks to use different versions of the same DLL using WithCustomBuildConfiguration. This works pretty well. However, there's one gotcha that I don't understand: it appears to always use RyuJIT DEBUG.

It appears to have the effect that certain optimizations do not take place (which is not surprising if it is indeed the RyuJIT DEBUG running). However, I'm not 100% certain if this is just a flaw in the reporting, because when running each job separately (and then it shows just RyuJIT), the numbers are within error margin.

Anyway, this is the output I have:

// * Summary *

BenchmarkDotNet=v0.12.1, OS=Windows 7 SP1 (6.1.7601.0)
Intel Xeon CPU X5680 3.33GHz, 2 CPU, 12 logical and 12 physical cores
Frequency=3247138 Hz, Resolution=307.9635 ns, Timer=TSC
.NET Core SDK=3.1.400-preview-015151
  [Host]    : .NET Core 3.1.3 (CoreCLR 4.700.20.11803, CoreFX 4.700.20.12001), X64 RyuJIT DEBUG
  Before    : .NET Core 3.1.3 (CoreCLR 4.700.20.11803, CoreFX 4.700.20.12001), X64 RyuJIT
  Optimized : .NET Core 3.1.3 (CoreCLR 4.700.20.11803, CoreFX 4.700.20.12001), X64 RyuJIT DEBUG

MaxRelativeError=0.01  IterationTime=150.0000 ms  WarmupCount=1

Note that for my "Optimized" job I would expect it to use the same 64 bit RyuJIT. This is the config that I use (with help from @adamsitnik, from here: #1486). It's F#, but you'll get the idea:

let baseJob = Job.Default
                .WithWarmupCount(1) // 1 warmup is enough for our purpose
                .WithIterationTime(TimeInterval.FromMilliseconds(250.0)) // the default is 0.5s per iteration
                .WithMaxRelativeError(0.01)

let jobBefore = baseJob.WithId("Before").WithBaseline(true)

let jobAfter = baseJob.WithCustomBuildConfiguration("LocalBuild").WithId("Optimized")

let config = DefaultConfig.Instance.AddJob(jobBefore).AddJob(jobAfter).KeepBenchmarkFiles()

BenchmarkRunner.Run<BenchToString>(config)
|> ignore

[adamsitnik]

the fix has been provided in #1494

a workaround for now:

create an MsBuild argument and add it to the job using this method:

var arg = new MsBuildArgument("/p:Optimize=true");
job = job.WithArguments(new [] { arg });

[abelbraaksma]

Awesome! And thanks for the workaround!