The dotnet store is incorrectly including BCL assemblies when targeting .NET Core 3.1

[normj]

Using the following store manifest that only includes Newtonsoft.Json

<Project Sdk="Microsoft.NET.Sdk">
  <ItemGroup>
    <PackageReference Include="Newtonsoft.Json" Version="9.0.1" />
  </ItemGroup>
</Project>

I execute the following command to create a store for .NET Core 2.1:

dotnet store --manifest .\manifest.xml -f netcoreapp2.1 -o ./store2.1 -r linux-x64 --skip-optimization

the store artifact.xml contains only Newtonsoft.Json as expected.

<StoreArtifacts>
  <Package Id="Newtonsoft.Json" Version="9.0.1" />
</StoreArtifacts>

But then creating the store for .NET Core 3.1 using the command:

dotnet store --manifest .\manifest.xml -f netcoreapp3.1 -o ./store3.1 -r linux-x64 --skip-optimization

Creates a store with a lot of runtime and system dlls which bloat the package store and also those assembles fail to the store optimization.

<StoreArtifacts>
  <Package Id="Microsoft.CSharp" Version="4.0.1" />
  <Package Id="Newtonsoft.Json" Version="9.0.1" />
  <Package Id="runtime.any.System.Collections" Version="4.0.11" />
  <Package Id="runtime.any.System.Diagnostics.Tools" Version="4.0.1" />
  <Package Id="runtime.any.System.Globalization" Version="4.0.11" />
  <Package Id="runtime.any.System.IO" Version="4.1.0" />
  <Package Id="runtime.any.System.Reflection" Version="4.1.0" />
  <Package Id="runtime.any.System.Reflection.Extensions" Version="4.0.1" />
  <Package Id="runtime.any.System.Reflection.Primitives" Version="4.0.1" />
  <Package Id="runtime.any.System.Resources.ResourceManager" Version="4.0.1" />
  <Package Id="runtime.any.System.Runtime" Version="4.1.0" />
  <Package Id="runtime.any.System.Runtime.Handles" Version="4.0.1" />
  <Package Id="runtime.any.System.Runtime.InteropServices" Version="4.1.0" />
  <Package Id="runtime.any.System.Text.Encoding" Version="4.0.11" />
  <Package Id="runtime.any.System.Text.Encoding.Extensions" Version="4.0.11" />
  <Package Id="runtime.any.System.Threading.Tasks" Version="4.0.11" />
  <Package Id="runtime.unix.System.Diagnostics.Debug" Version="4.0.11" />
  <Package Id="runtime.unix.System.IO.FileSystem" Version="4.0.1" />
  <Package Id="runtime.unix.System.Private.Uri" Version="4.0.1" />
  <Package Id="runtime.unix.System.Runtime.Extensions" Version="4.1.0" />
  <Package Id="System.Dynamic.Runtime" Version="4.0.11" />
  <Package Id="System.IO.FileSystem.Primitives" Version="4.0.1" />
  <Package Id="System.Linq" Version="4.1.0" />
  <Package Id="System.Linq.Expressions" Version="4.1.0" />
  <Package Id="System.ObjectModel" Version="4.0.12" />
  <Package Id="System.Reflection.Emit" Version="4.0.1" />
  <Package Id="System.Reflection.Emit.ILGeneration" Version="4.0.1" />
  <Package Id="System.Reflection.Emit.Lightweight" Version="4.0.1" />
  <Package Id="System.Reflection.TypeExtensions" Version="4.1.0" />
  <Package Id="System.Runtime.Serialization.Primitives" Version="4.1.1" />
  <Package Id="System.Text.RegularExpressions" Version="4.1.0" />
  <Package Id="System.Threading" Version="4.0.11" />
  <Package Id="System.Threading.Tasks.Extensions" Version="4.0.0" />
  <Package Id="System.Xml.ReaderWriter" Version="4.0.11" />
  <Package Id="System.Xml.XDocument" Version="4.0.11" />
</StoreArtifacts>

This change in behavior seems to make the dotnet store command a broken experience.

@nguerrera We have talked about dotnet store in the past. Can you shed any light on why this is happening now?

[normj]

Friendly ping on this to get some attention. This is breaking some of our scenarios for .NET Core 3.1.

[rpopovych]

This is a bummer. Prevents us from using 3.1 in some of our AWS Lambdas. Hoping there is an easy resolution/workaround for this issue.

[davidruhmann]

Wonder if the root cause is related to this issue dotnet/runtime#2836;

Found the issue referenced in the code to handle an additional file being stored. https://github.com/dotnet/sdk/blob/master/src/Tests/Microsoft.NET.Publish.Tests/GivenThatWeWantToStoreAProjectWithDependencies.cs#L92

From a quick look at different versions, this only seems to happen for Newtonsoft.Json versions < 11. Only major change that I initially see to cause this is that version 11 starts referencing netstandard2.0.

[dsplaisted]

This is a result of changes we made in .NET Core 3. We no longer use PackageReferences to represent .NET Core Framework. This is discussed here: dotnet/designs#50

The old version of NewtonSoft.Json has dependencies to packages which are part of .NET Core. Because we don't represent .NET Core as a PackageReference anymore, the dotnet store logic no longer knows to exclude them from the store.

The ideal fix to this would likely be NuGet/Home#7344, and possibly additional changes in dotnet store to use it.

Possible workarounds:

• Use a later version of NewtonSoft.Json (or in general, use packages that don't have these older dependencies, which will generally be the case if the package was built for .NET Core 2.0 or .NET Standard 2.0 or higher).
• Exclude each unneeded package from the manifest by adding the following

  <Target Name="AddPackagesToPrune" AfterTargets="PrepforRestoreForComposeStore">
    <ItemGroup>
      <PackagesToPrune Include="Microsoft.CSharp" />
      <PackagesToPrune Include="System.Collections" />
      <PackagesToPrune Include="System.Diagnostics.Debug" />
      <PackagesToPrune Include="System.Dynamic.Runtime" />
      <PackagesToPrune Include="System.Globalization" />
      <PackagesToPrune Include="System.IO" />
      <PackagesToPrune Include="System.Linq" />
      <PackagesToPrune Include="System.Linq.Expressions" />
      <PackagesToPrune Include="System.ObjectModel" />
      <PackagesToPrune Include="System.Reflection" />
      <PackagesToPrune Include="System.Reflection.Extensions" />
      <PackagesToPrune Include="System.Resources.ResourceManager" />
      <PackagesToPrune Include="System.Runtime" />
      <PackagesToPrune Include="System.Runtime.Extensions" />
      <PackagesToPrune Include="System.Runtime.Serialization.Primitives" />
      <PackagesToPrune Include="System.Text.Encoding" />
      <PackagesToPrune Include="System.Text.Encoding.Extensions" />
      <PackagesToPrune Include="System.Text.RegularExpressions" />
      <PackagesToPrune Include="System.Threading" />
      <PackagesToPrune Include="System.Threading.Tasks" />
      <PackagesToPrune Include="System.Xml.ReaderWriter" />
      <PackagesToPrune Include="System.Xml.XDocument" />
    </ItemGroup>
    <PropertyGroup>
      <PackagesToPrune>$(PackagesToPrune);@(PackagesToPrune)</PackagesToPrune>
    </PropertyGroup>
  </Target>

Note that if there is a PackageToPrune which is not part of the dependency graph, you will get a NullReferenceException from one of the tasks.

[normj]

Interesting. In my case dotnet store is used as a general tool and I don't have control what packages are in the manifest. I happen to pick Newtonsoft.Json as an example because one of my libraries has a dependency on that version that is commonly put into the store along with the customers packages.

I have the option to make a modifications to a copy of the inputted manifest and use that when calling dotnet store. I need to come up with a way given a manifest be able to compute what packages to prune or at a minimum detect that package will bring in framework packages and they should upgrade their package versions.

[dsplaisted]

@normj Could you update your library which depends on NewtonSoft.Json to depend on at least 11.0.1? That should help mitigate the issue.

[normj]

I can, but I can't control whether somebody uses our tool that relies dotnet store with dependencies that will also have this problem. Is there anyway for me to programmatically detect when a store manifest would potentially bring in these bad refs? I'm afraid I would need to build the whole dependency graph myself first and inspect before calling dotnet store.

[davidruhmann]

@dsplaisted, do you see any issues with this type of workaround until NuGet/Home#7344 is resolved?

• Run dotnet store on the initial manifest.
• Create a new manifest file with a prune list (consisting of entries of dependencies in the artifact.xml that also exist in the runtime folder (e.g. /usr/local/share/dotnet/shared/Microsoft.NETCore.App/3.1.3)
  • from System.Runtime.InteropServices.RuntimeEnvironment.GetRuntimeDirectory() but for the target framework.
• Delete the old output.
• Then run dotnet store again on the new manifest file.

Am I missing any major aspect regarding this?

[dsplaisted]

@davidruhmann I think something like that could work, but mapping from the dependencies in the runtime folder to packages that should be excluded may not be simple. You have to consider not just the file name but the version number, and also map it to the right NuGet package.

[davidruhmann]

Appreciate the quick reply!

Version did cross my mind, but wouldn't that just be for higher versioned references? Which, if I understand correctly, could still be resolved by the store consumer? Meaning the store would possibly be missing some elements, but not the flip side of including unnecessary elements.

As for the naming of nuget packages to BCL components, are there many differences?

I am thinking of this workaround in a scope of 80% success or something like that. If it can solve the majority of conflicts for the current time until NuGet/Home#7344 is complete; that should be good enough vs making the workaround too complex. Even if the other 20% result in failure, the original bloated store could still be used or fail the process entirely saying unsupported.

[clearwaterstream]

@davidruhmann I must admit I have not patience for gardening, but pruning I did try. Initial manifest -> dotnet store -> look at resulting artifact.xml -> add the extraneous packages to prune list on initial manifest -> dotnet store again.

What I found is that the AddPackagesToPrune would choke on certain packages. Let's do a simple example.

Initial manifest:

<Project Sdk="Microsoft.NET.Sdk.Web">
  <ItemGroup>
	<PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.1.3" />
  </ItemGroup>
</Project>

dotnet store --manifest initial.xml -f netcoreapp3.1 -o bin/dotnetcore/store -r linux-x64 --skip-optimization

Output is:

<StoreArtifacts>
  <Package Id="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.1.3" />
  <Package Id="Microsoft.CSharp" Version="4.3.0" />
  <Package Id="Microsoft.IdentityModel.JsonWebTokens" Version="5.5.0" />
  <Package Id="Microsoft.IdentityModel.Logging" Version="5.5.0" />
  <Package Id="Microsoft.IdentityModel.Protocols" Version="5.5.0" />
  <Package Id="Microsoft.IdentityModel.Protocols.OpenIdConnect" Version="5.5.0" />
  <Package Id="Microsoft.IdentityModel.Tokens" Version="5.5.0" />
  <Package Id="Newtonsoft.Json" Version="10.0.1" />
  <Package Id="runtime.any.System.Collections" Version="4.3.0" />
  <Package Id="runtime.any.System.Diagnostics.Tools" Version="4.3.0" />
  <Package Id="runtime.any.System.Diagnostics.Tracing" Version="4.3.0" />
  <Package Id="runtime.any.System.Globalization" Version="4.3.0" />
  <Package Id="runtime.any.System.IO" Version="4.3.0" />
  <Package Id="runtime.any.System.Reflection" Version="4.3.0" />
  <Package Id="runtime.any.System.Reflection.Extensions" Version="4.3.0" />
  <Package Id="runtime.any.System.Reflection.Primitives" Version="4.3.0" />
  <Package Id="runtime.any.System.Resources.ResourceManager" Version="4.3.0" />
  <Package Id="runtime.any.System.Runtime" Version="4.3.0" />
  <Package Id="runtime.any.System.Runtime.Handles" Version="4.3.0" />
  <Package Id="runtime.any.System.Runtime.InteropServices" Version="4.3.0" />
  <Package Id="runtime.any.System.Text.Encoding" Version="4.3.0" />
  <Package Id="runtime.any.System.Text.Encoding.Extensions" Version="4.3.0" />
  <Package Id="runtime.any.System.Threading.Tasks" Version="4.3.0" />
  <Package Id="runtime.unix.System.Diagnostics.Debug" Version="4.3.0" />
  <Package Id="runtime.unix.System.IO.FileSystem" Version="4.3.0" />
  <Package Id="runtime.unix.System.Private.Uri" Version="4.3.0" />
  <Package Id="runtime.unix.System.Runtime.Extensions" Version="4.3.0" />
  <Package Id="System.Buffers" Version="4.3.0" />
  <Package Id="System.Collections.NonGeneric" Version="4.3.0" />
  <Package Id="System.Collections.Specialized" Version="4.3.0" />
  <Package Id="System.ComponentModel" Version="4.3.0" />
  <Package Id="System.ComponentModel.Primitives" Version="4.3.0" />
  <Package Id="System.ComponentModel.TypeConverter" Version="4.3.0" />
  <Package Id="System.Dynamic.Runtime" Version="4.3.0" />
  <Package Id="System.Globalization.Extensions" Version="4.3.0" />
  <Package Id="System.IdentityModel.Tokens.Jwt" Version="5.5.0" />
  <Package Id="System.IO.FileSystem.Primitives" Version="4.3.0" />
  <Package Id="System.Linq" Version="4.3.0" />
  <Package Id="System.Linq.Expressions" Version="4.3.0" />
  <Package Id="System.ObjectModel" Version="4.3.0" />
  <Package Id="System.Reflection.Emit" Version="4.3.0" />
  <Package Id="System.Reflection.Emit.ILGeneration" Version="4.3.0" />
  <Package Id="System.Reflection.Emit.Lightweight" Version="4.3.0" />
  <Package Id="System.Reflection.TypeExtensions" Version="4.3.0" />
  <Package Id="System.Runtime.Numerics" Version="4.3.0" />
  <Package Id="System.Runtime.Serialization.Formatters" Version="4.3.0" />
  <Package Id="System.Runtime.Serialization.Primitives" Version="4.3.0" />
  <Package Id="System.Security.Cryptography.Cng" Version="4.5.0" />
  <Package Id="System.Text.RegularExpressions" Version="4.3.0" />
  <Package Id="System.Threading" Version="4.3.0" />
  <Package Id="System.Threading.Tasks.Extensions" Version="4.3.0" />
  <Package Id="System.Xml.ReaderWriter" Version="4.3.0" />
  <Package Id="System.Xml.XDocument" Version="4.3.0" />
  <Package Id="System.Xml.XmlDocument" Version="4.3.0" />
</StoreArtifacts>

initial.xml with one package added to prune:

<Project Sdk="Microsoft.NET.Sdk.Web">
  <ItemGroup>
	<PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.1.3" />
  </ItemGroup>
  <Target Name="AddPackagesToPrune" AfterTargets="PrepforRestoreForComposeStore">
    <ItemGroup>
      <PackagesToPrune Include="Microsoft.CSharp" />
    </ItemGroup>
    <PropertyGroup>
      <PackagesToPrune>$(PackagesToPrune);@(PackagesToPrune)</PackagesToPrune>
    </PropertyGroup>
  </Target>
</Project>

Prune worked -- there are less packages in artifact.xml. Great.

Now, if I do this:

<Project Sdk="Microsoft.NET.Sdk.Web">
  <ItemGroup>
	<PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="3.1.3" />
	<PackageReference Include="Amazon.Lambda.AspNetCoreServer" Version="5.0.0" />
  </ItemGroup>
  <Target Name="AddPackagesToPrune" AfterTargets="PrepforRestoreForComposeStore">
    <ItemGroup>
      <PackagesToPrune Include="Microsoft.CSharp" />
    </ItemGroup>
    <PropertyGroup>
      <PackagesToPrune>$(PackagesToPrune);@(PackagesToPrune)</PackagesToPrune>
    </PropertyGroup>
  </Target>
</Project>

I will get Microsoft.NET.ComposeStore.targets(420,5): error MSB4018: The "FilterResolvedFiles" task failed unexpectedly. error.

[davidruhmann]

@clearwaterstream Thanks for creating the proof of concept. I see, this error is because the Filter target is being run per dependency and the Amazon dependency is not pulling in Microsoft.CSharp. Have to step away, but wonder if we can inject in before the Filter target to dynamically build the prune list per dependency. Will take a look later but it will most likely hinge on how the FilterResolvedFiles target is written and if ResolvedFiles is accessible to a before target.
https://github.com/dotnet/sdk/blob/master/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.ComposeStore.targets#L420