MSBuild parameters are not passed to generated benchmark project

[xplicit]

I run the following command to build and run benchmarks:

dotnet restore /p:BASELINE=1
dotnet build /p:BASELINE=1 -c Release
dotnet exec -c Release -f netcoreapp1.1 bin/Release/netcoreapp1.1/benchmarks.dll

But BenchmarkDotNet creates its own wrapping project and does not pass /p:BASELINE=1 to msbuild. Here is content of BDN.Generated.sh

call dotnet restore --no-dependencies
call dotnet build --framework netcoreapp1.1 --configuration Release

while it should be

call dotnet restore /p:BASELINE=1 --no-dependencies
call dotnet build /p:BASELINE=1 --framework netcoreapp1.1 --configuration Release

[adamsitnik]

Hello @xplicit !

Thanks for a very well described problem!

We have a feature that allows using parameters in our benchmarks. It's called params, here you can find full description.

Could you explain me your motivation behind passing custom parameters as console app parameters to the generated process? Or maybe our [Params] feature can be enough for you?

[xplicit]

The idea of passing msbuild params is to make benchmarks for different assembly versions of the same library. When I pass /p:BASELINE=1 I compile and run benchmarks with baseline version of lib, when I did not pass it I compile and run benchmarks with current code of library.

So the high-level idea is to run v1 of mylib.dll and v2 of mylib.dll and see if there are performance regressions during the code changes.

[AndreyAkinshin]

In the future, I want to implement a feature which allows setting custom environment variable in each job, it will help to support features like this: #262
@adamsitnik, maybe it makes sense to introduce also an ability to setting build arguments?

[adamsitnik]

@xplicit the idea sounds very cool! But I can't find any information about this parameter. Could you provide me some link? I would like to implement and test it.

[xplicit]

@adamsitnik BASELINE is custom property which I define from msbuild command line using /p parameter (see https://msdn.microsoft.com/ru-ru/library/ms164311.aspx /property parameter). Then I use it in conditional csproj. Something like that:

  <Choose>
    <When Condition=" '$(BASELINE)' == '' ">
      <ItemGroup>
        <ProjectReference Include="..\..\src\ServiceStack.Text\ServiceStack.Text.csproj" Version="1.0.*" />
      </ItemGroup>
    </When>
    <When Condition=" '$(BASELINE)' != '' ">
      <ItemGroup Condition=" '$(TargetFramework)' == 'net46' ">
        <PackageReference Include="ServiceStack.Text" Version="4.5.8" />
      </ItemGroup>
      <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
        <PackageReference Include="ServiceStack.Text.Core" Version="1.0.38" />
      </ItemGroup>
    </When>
  </Choose>

So when I pass /p:BASELINE=1 into msbuild I link with "baseline" version of library, when I do not pass it I link with current developing project.

[adamsitnik]

@xplicit now I get it, thanks! I will try to implement it this week

my long term plan is to add similar feature to BenchmarkDotNet. It could be sth like this:

Job.Default.WithReferences(new PackageReference("PackageName", "1.0.0"));
Job.Default.WithReferences(new PackageReference("PackageName", "2.0.0"));
Job.Default.WithReferences(new ProjectReference("pathToCsProj"));

I wanted to implement it a long time ago, but then project.json was being abandoned and I wanted to wait.