Test a benchmark against different versions of the same assembly, one not in NuGet, or both local

[abelbraaksma]

When working on optimizations on NuGet assemblies, the new version often isn't yet in NuGet. In my case, I'm working on optimizations of FSharp.Core.dll, and because of syntax differences in the way certain methods are created (syntax that is allowed inside FSharp.Core but not in writing regular user code), I cannot easily create copies of the members that I want to compare.

My current way of doing this is to use Reflection.Emit and friends with AssemblyBuilder, but this is complex and leads to potential side effects that are hard to optimize away.

I can of course simply create two separate projects and compare the results. But I was wondering, considering this PR #922, whether there's a better/easier way, or built-in way of doing the same.

[adamsitnik]

Hi @abelbraaksma

BenchmarkDotNet generates a new project on the fly. This is the template that we are using for .NET Core: https://github.com/dotnet/BenchmarkDotNet/blob/master/src/BenchmarkDotNet/Templates/CsProj.txt

There are ways of controlling MsBuild's behavior from the BDN config level like using a custom build configuration or passing property to MsBuild.

Could you please provide your project file that is running F# app against a local build of FSharp.Core.dll? Once I have it I should be able to help.

[abelbraaksma]

@adamsitnik, it took me a while to get back to you, busy week, but now it's weekend ;).

I placed a test-solution here: https://github.com/abelbraaksma/StringLength-Optimization, it's very simple and probably self-explanatory.

Thinking a bit more about this, I don't think it is currently possible, because (1) this is F# specific, and F# has special inlining rules on certain functions from the FSharp.Core.dll and (2) it requires switching <DisableImplicitFSharpCoreReference> on or off, which isn't available for C# afaik.

Note that I tried using a "normal" reference, but without the above special fsproj declaration, the normal reference will not be used by the compiler (for inlining), nor for the final reference. |As a result, even manually copying the Core dll to the output dir won't help.

Hopefully you have a creative idea, it would be nice if I could make this work for future perf tests.

[adamsitnik]

@abelbraaksma Thanks for providing a nice sample! I was able to get it working and sent a PR to your fork: abelbraaksma/StringLength-Optimization#2

If anything does not work as expected or something is not clear please feel free to reopen this issue|ask more questions.

Thanks!