Can't benchmark library that targets 'net5.0-windows' framework

[jonstelly]

I've got a library that has some WPF components so its target framework is net5.0-windows. So my unit tests / benchmark project also has to target net5.0-windows. I'm trying to run benchmarkdotnet from xUnit using:

[SimpleJob(RuntimeMoniker.NetCoreApp50)]
[SimpleJob(RuntimeMoniker.CoreRt50)]
public class BitmapBenchmarks
{
    [Benchmark]
    public void FindVisibleMarker()
    {
        //My benchmark test stuff here...
    }
}

[Fact]
public void BitmapBenchmarks()
{
    var summary = BenchmarkRunner.Run<BitmapBenchmarks>();
    Logger.Information("Results: ...", summary...);
}

and I'm getting this error:

error NU1201: Project MyThing.Tests is not compatible with net5.0 (.NETCoreApp,Version=v5.0). Project MyThing.Tests supports: net5.0-windows7.0 (.NETCoreApp,Version=v5.0)

I can't change the benchmark project to net5.0 because the underlying library targets net5.0-windows. Is there some way to run these benchmarks against the net50-windows framework?

I know net50 is still in preview, so I'll completely understand if this just isn't supported yet.

[adamsitnik]

Hi @jonstelly

I was able to repro the problem you have hit.

using BenchmarkDotNet.Attributes;

namespace repro
{
    class Program
    {
        static void Main(string[] args) => BenchmarkDotNet.Running.BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);
    }

    public class Benchmarks
    {
        [Benchmark] public void Empt() { }
    }
}

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net5.0-windows</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.12.1" />
  </ItemGroup>
</Project>

let me try to find a workaround and a fix

[adamsitnik]

@jonstelly before I provide a proper fix I have the following workaround:

it's possible to build a BDN config that uses provided target framework moniker. You need to use it and pass to BenchmarkRunner or BenchmarkSwitcher:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;
using BenchmarkDotNet.Toolchains.CsProj;
using BenchmarkDotNet.Toolchains.DotNetCli;

namespace repro
{
    class Program
    {
        static void Main(string[] args)
        {
            var config = DefaultConfig.Instance
                .AddJob(
                    Job.Default.WithToolchain(
                        CsProjCoreToolchain.From(
                            new NetCoreAppSettings(
                                targetFrameworkMoniker: "net5.0-windows",  // the key to make it work
                                runtimeFrameworkVersion: null,
                                name: "5.0")))
                                .AsDefault());

            BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args, config);
        }
    }

    public class Benchmarks
    {
        [Benchmark] public void Empt() { }
    }
}

But due to #1479 it won't work with the current latest version available at nuget.org (0.12.1). You also need to use a NuGet package from our CI feed and latest version to be able to make it work (it's going to change as soon as we release 0.12.2):

nuget.config

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
      <add key="bdn-ci" value="https://ci.appveyor.com/nuget/benchmarkdotnet" />
  </packageSources>
</configuration>

.csproj

  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.12.1.1420" />
  </ItemGroup>

[adamsitnik]

I currently don't have a good idea for a proper fix. The problem is that the .dll that defines benchmarks and was built for net5.0-windows does not contain any information that indicates that and could be read using reflection:

I am going to ask a question in the SDK repo

[adamsitnik]

I've asked a question in the SDK repo, as soon as I get an answer that allows me to implement a fix, I am going to fix it.

dotnet/sdk#13668

[adamsitnik]

@AndreyAkinshin I've set the 0.12.2 milestone to make sure that we include this in the next release and have a full .NET 5 support.

[jonstelly]

Thanks a lot for getting to this so quickly. I'll test out the CI packages and custom moniker config sometime this week.