Mysterious type/namespace errors in release/3.1

[wtgodbe]

Errors like the following started popping up in release/3.1 on April 6:

FormPipeReader.cs(9,17): error CS0234: The type or namespace name 'Pipelines' does not exist in the namespace 'System.IO' (are you missing an assembly reference?)

The last green build was on 5019088

The first red build was on 31203f0

As the failures started on a no-op merge PR, this is very confusing. It may be related to a recent update to the VS versions on the build machines, or it may not.

Info:

• It repros on VS 16.5.0 & 16.5.1, but not 16.4.5
• Adding rollForward: major and updating the VS version in global.json didn't fix it in CI
• Experimental build confirms that it's not due to a code change, but the VS Rollout: https://dev.azure.com/dnceng/internal/_build/results?buildId=594826
• It affects only non-packable projects in the shared framework
• It happens when a test project is building its project references (One example being IIS.NewHandler.FunctionalTests)

Still digging, but this is a blocker for 3.1.4 and needs to be fixed by Tuesday

CC @dotnet/aspnet-build

[wtgodbe]

Full error messages:

FormPipeReader.cs(9,17): error CS0234: The type or namespace name 'Pipelines' does not exist in the namespace 'System.IO' (are you missing an assembly reference?) [F:\workspace_work\1\s\src\Http\WebUtilities\src\Microsoft.AspNetCore.WebUtilities.csproj]
FormPipeReader.cs(42,31): error CS0246: The type or namespace name 'PipeReader' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Http\WebUtilities\src\Microsoft.AspNetCore.WebUtilities.csproj]
FormPipeReader.cs(47,31): error CS0246: The type or namespace name 'PipeReader' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Http\WebUtilities\src\Microsoft.AspNetCore.WebUtilities.csproj]
FormPipeReader.cs(39,26): error CS0246: The type or namespace name 'PipeReader' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Http\WebUtilities\src\Microsoft.AspNetCore.WebUtilities.csproj]
SocketTransportFactory.cs(12,28): error CS0234: The type or namespace name 'Options' does not exist in the namespace 'Microsoft.Extensions' (are you missing an assembly reference?) [F:\workspace_work\1\s\src\Servers\Kestrel\Transport.Sockets\src\Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.csproj]
SocketTransportFactory.cs(22,13): error CS0246: The type or namespace name 'IOptions<>' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Servers\Kestrel\Transport.Sockets\src\Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.csproj]

GenericHost\GenericWebHostedService.cs(21,28): error CS0234: The type or namespace name 'Options' does not exist in the namespace 'Microsoft.Extensions' (are you missing an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
Http\DefaultHttpContextFactory.cs(9,28): error CS0234: The type or namespace name 'Options' does not exist in the namespace 'Microsoft.Extensions' (are you missing an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
WebHostBuilderExtensions.cs(118,106): error CS0246: The type or namespace name 'ServiceProviderOptions' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
WebHostBuilderExtensions.cs(129,129): error CS0246: The type or namespace name 'ServiceProviderOptions' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
WebHostBuilderExtensions.cs(166,97): error CS0246: The type or namespace name 'ILoggingBuilder' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
WebHostBuilderExtensions.cs(177,120): error CS0246: The type or namespace name 'ILoggingBuilder' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
GenericHostWebHostBuilderExtensions.cs(9,58): error CS0246: The type or namespace name 'IHostBuilder' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
GenericHostWebHostBuilderExtensions.cs(9,23): error CS0246: The type or namespace name 'IHostBuilder' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
GenericHost\GenericWebHostApplicationLifetime.cs(13,26): error CS0246: The type or namespace name 'IHostApplicationLifetime' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
GenericHost\GenericWebHostApplicationLifetime.cs(14,50): error CS0246: The type or namespace name 'IHostApplicationLifetime' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
GenericHost\GenericWebHostBuilder.cs(186,88): error CS0246: The type or namespace name 'ServiceProviderOptions' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
GenericHost\GenericWebHostBuilder.cs(214,51): error CS0246: The type or namespace name 'HostBuilderContext' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
GenericHost\GenericWebHostBuilder.cs(291,53): error CS0246: The type or namespace name 'HostBuilderContext' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
GenericHost\GenericWebHostBuilder.cs(312,64): error CS0246: The type or namespace name 'HostBuilderContext' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
GenericHost\ISupportsUseDefaultServiceProvider.cs(11,81): error CS0246: The type or namespace name 'ServiceProviderOptions' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
GenericHost\GenericWebHostBuilder.cs(23,26): error CS0246: The type or namespace name 'IHostBuilder' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
GenericHost\GenericWebHostBuilder.cs(30,38): error CS0246: The type or namespace name 'IHostBuilder' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
GenericHost\GenericWebHostedService.cs(27,44): error CS0246: The type or namespace name 'IHostedService' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
Internal\HostedServiceExecutor.cs(34,46): error CS0246: The type or namespace name 'IHostedService' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
Internal\HostedServiceExecutor.cs(15,38): error CS0246: The type or namespace name 'IHostedService' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
Internal\HostedServiceExecutor.cs(18,89): error CS0246: The type or namespace name 'IHostedService' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
Internal\HostingEnvironment.cs(9,81): error CS0234: The type or namespace name 'IHostingEnvironment' does not exist in the namespace 'Microsoft.Extensions.Hosting' (are you missing an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
GenericHost\GenericWebHostedService.cs(29,38): error CS0246: The type or namespace name 'IOptions<>' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
Internal\HostingEnvironment.cs(9,20): error CS0012: The type 'IHostEnvironment' is defined in an assembly that is not referenced. You must add a reference to assembly 'Microsoft.Extensions.Hosting.Abstractions, Version=3.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
GenericHost\HostingStartupWebHostBuilder.cs(64,88): error CS0246: The type or namespace name 'ServiceProviderOptions' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
Internal\ApplicationLifetime.cs(15,83): error CS0234: The type or namespace name 'IApplicationLifetime' does not exist in the namespace 'Microsoft.Extensions.Hosting' (are you missing an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
Internal\ApplicationLifetime.cs(15,105): error CS0246: The type or namespace name 'IHostApplicationLifetime' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
StaticWebAssets\StaticWebAssetsFileProvider.cs(40,16): error CS0246: The type or namespace name 'PhysicalFileProvider' could not be found (are you missing a using directive or an assembly reference?) [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
Internal\HostingEnvironment.cs(9,41): error CS0012: The type 'IHostEnvironment' is defined in an assembly that is not referenced. You must add a reference to assembly 'Microsoft.Extensions.Hosting.Abstractions, Version=3.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]
Internal\HostingEnvironment.cs(9,102): error CS0012: The type 'IHostEnvironment' is defined in an assembly that is not referenced. You must add a reference to assembly 'Microsoft.Extensions.Hosting.Abstractions, Version=3.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. [F:\workspace_work\1\s\src\Hosting\Hosting\src\Microsoft.AspNetCore.Hosting.csproj]

[wtgodbe]

Oh, wait a minute - the pipelines issue makes it seem like it's related to that ref pack issue in 3.1.2. Maybe 16.5.1 uses the 3.1.2 runtime, and we got bit by our own bug 😆. Will investigate.

[wtgodbe]

Ok, here's what I think is happening: The build machines just updated to VS 16.5.1, which contains SDK 3.1.200, which contains runtime 3.1.2, which has this bug: #19133. That would explain why building a project locally doesn't fail the build, but building a solution does - our global.json lists the 3.1.103 SDK, which doesn't have the bug - but when building a .sln, VS uses the SDK it knows about, which does have the bug. The confusing this is that the suggested workaround is to add a direct PackageReference to the affected libraries (e.g. System.IO.Pipelines), but our affected project already has that:

aspnetcore/src/Http/WebUtilities/src/Microsoft.AspNetCore.WebUtilities.csproj

Lines 19 to 22 in 31203f0

	<ItemGroup>
	<Reference Include="Microsoft.Net.Http.Headers" />
	<Reference Include="System.IO.Pipelines" />
	</ItemGroup>

(That Reference gets converted to a PackageReference by our build system). Not 100% sure what to do yet - there is currently no build pool that has a 16.4 or 16.5.2 version of VS that we could use. Still brainstorming.

FYI @Pilchie since this is a blocker for 3.1.4

[Pilchie]

Can we look at a binlog and see what's happening to the PackageReference? It's clearly not making it through into the set of references passed to csc, so something is presumably knocking it out of the dependency graph.

[wtgodbe]

Can we look at a binlog and see what's happening to the PackageReference? It's clearly not making it through into the set of references passed to csc, so something is presumably knocking it out of the dependency graph.

I'm grabbing one from a build now. Sadly I haven't been able to repro locally since I can't install a specific version of VS, and the latest 16.5 already has an SDK with the bug patched

[wtgodbe]

Good call, the only reference to pipes passed to CSC is reference:F:\workspace\_work\1\s\.dotnet\packs\Microsoft.NETCore.App.Ref\3.1.0\ref\netcoreapp3.1\System.IO.Pipes.dll. So we must be resolving that reference to a framework reference. I'll try adding an explicit package ref there, and try to figure out what refs are missing in our other projects.

[wtgodbe]

#20760 will fix in 3.1.5