crossgen2 doesn't resolve compilers for non-portable rids

[tmds]

I'm trying to make runtime repo build using artifacts produced for a non-portable rid in dotnet/runtime#75597.

The build fails while trying to resolve the R2R compiler:

  13:47:14.530    19>Target "ResolveReadyToRunCompilers: (TargetId:2247)" in file "/home/tmds/sdk/sdk/7.0.100-rtm.22478.1/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Publish.targets" from project "/home/tmds/tarball41/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj" (target "_PrepareForReadyToRunCompilation" depends on it):
                     Using "ResolveReadyToRunCompilers" task from assembly "/home/tmds/sdk/sdk/7.0.100-rtm.22478.1/Sdks/Microsoft.NET.Sdk/targets/../tools/net7.0/Microsoft.NET.Build.Tasks.dll".
                     Task "ResolveReadyToRunCompilers" (TaskId:1425)
                       Task Parameter:ReadyToRunUseCrossgen2=True (TaskId:1425)
                       Task Parameter:
                           TargetingPacks=
                               Microsoft.NETCore.App
                                       NuGetPackageId=Microsoft.NETCore.App.Ref
                                       NuGetPackageVersion=7.0.0-rtm.22465.3
                                       PackageConflictPreferredPackages=Microsoft.NETCore.App.Ref;Microsoft.NETCore.App.Runtime.fedora.36-x64
                                       PackageDirectory=/home/tmds/sdk/packs/Microsoft.NETCore.App.Ref/7.0.0-rtm.22465.3
                                       Path=/home/tmds/sdk/packs/Microsoft.NETCore.App.Ref/7.0.0-rtm.22465.3
                                       RuntimeFrameworkName=Microsoft.NETCore.App
                                       RuntimePackRuntimeIdentifiers=fedora.36-x64
                                       TargetFramework=net7.0
                                       TargetingPackFormat= (TaskId:1425)
                       Task Parameter:NETCoreSdkRuntimeIdentifier=fedora.36-x64 (TaskId:1425)
                       Task Parameter:
                           Crossgen2Packs=
                               Microsoft.NETCore.App.Crossgen2.fedora.36-x64
                                       NuGetPackageId=Microsoft.NETCore.App.Crossgen2.fedora.36-x64
                                       NuGetPackageVersion=7.0.0-rtm.22465.3
                                       OriginalItemSpec=Microsoft.NETCore.App.Crossgen2.fedora.36-x64
                                       PackageDirectory=/home/tmds/tarball41/src/runtime/artifacts/source-build/self/package-cache/microsoft.netcore.app.crossgen2.fedora.36-x64/7.0.0-rtm.22465.3 (TaskId:1425)
                       Task Parameter:
                           RuntimePacks=
                               Microsoft.NETCore.App.Runtime.fedora.36-x64
                                       FrameworkName=Microsoft.NETCore.App
                                       IsTrimmable=
                                       NuGetPackageId=Microsoft.NETCore.App.Runtime.fedora.36-x64
                                       NuGetPackageVersion=7.0.0-rtm.22465.3
                                       OriginalItemSpec=Microsoft.NETCore.App.Runtime.fedora.36-x64
                                       PackageDirectory=/home/tmds/tarball41/src/runtime/artifacts/source-build/self/package-cache/microsoft.netcore.app.runtime.fedora.36-x64/7.0.0-rtm.22465.3
                                       RuntimeIdentifier=fedora.36-x64 (TaskId:1425)
                       Task Parameter:PerfmapFormatVersion=1 (TaskId:1425)
                       Task Parameter:RuntimeGraphPath=/home/tmds/tarball41/src/runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/runtime.json (TaskId:1425)
  13:47:14.535    19>/home/tmds/sdk/sdk/7.0.100-rtm.22478.1/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Publish.targets(299,5): error NETSDK1095: Optimizing assemblies for performance is not supported for the selected target platform or architecture. Please verify you are using a supported runtime identifier, or set the PublishReadyToRun property to false. [/home/tmds/tarball41/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj]

Looking at the code in ResolveReadyToRunCompilers.cs, it looks like isSupportedTarget will be false because targetOS is will be null for a non-portable rid:

sdk/src/Tasks/Microsoft.NET.Build.Tasks/ResolveReadyToRunCompilers.cs

Lines 144 to 162 in b99ebbe

	bool version5 = crossgen2PackVersion.Major < 6;
	bool isSupportedTarget = ExtractTargetPlatformAndArchitecture(_targetRuntimeIdentifier, out _targetPlatform, out _targetArchitecture);
	string targetOS = _targetPlatform switch
	{
	"linux" => "linux",
	"linux-musl" => "linux",
	"osx" => "osx",
	"win" => "windows",
	_ => null
	};
	// In .NET 5 Crossgen2 supported only the following host->target compilation scenarios:
	// win-x64 -> win-x64
	// linux-x64 -> linux-x64
	// linux-musl-x64 -> linux-musl-x64
	isSupportedTarget = isSupportedTarget &&
	targetOS != null &&
	(!version5 || _targetRuntimeIdentifier == _hostRuntimeIdentifier) &&
	GetCrossgen2ComponentsPaths(version5);

cc @ViktorHofer @hoyosjs @AntonLapounov

How can we update this code so it works for non-portable rids?

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[AntonLapounov]

The passed RID should be normalized to one of the expected values as it is performed by SDK when you do, for example, dotnet publish -p:PublishReadyToRun=true -r:fedora.36-x64.

[tmds]

The SDK picks the best match. Because these packages were built from source, there's an exact match (fedora.36-x64) and it doesn't fall back to the portable rid (linux-x64).

[AntonLapounov]

We use NuGetUtils.GetBestMatchingRid to figure out the RID. Does it behave differently in source-build? If yes, can we emulate the usual behavior?

[tmds]

We use NuGetUtils.GetBestMatchingRid to figure out the RID. Does it behave differently in source-build? If yes, can we emulate the usual behavior?

If you call this function with a non-portable rid, and that non-portable rid is part of the available rids, then the function will give you back that rid. That makes sense, because it is the best match.

 string targetOS = _targetPlatform switch 
 { 
     "linux" => "linux", 
     "linux-musl" => "linux", 
     "osx" => "osx", 
     "win" => "windows", 
     _ => null 
 }; 

For my understanding: why is there a targetOS != null check?

[tmds]

Does it behave differently in source-build?

Yes it behaves different in source-build because we're building packages that don't have the Microsoft portable rids (like linux-x64), but non-portable packages named after the platform they represent (like fedora.36-x64).

If yes, can we emulate the usual behavior?

Currently we are emulating the usual behavior because source-build builds the runtime repo twice. Once it names everything linux-x64 so the rest of source-build can pretend those artifacts exist.

Building runtime repo twice costs a significant amount of time. We'd like to eliminate the 'portable runtime' build by making the repositories capable of consuming the non-portable artifacts. I hit this issue as part of dotnet/runtime#75597.

[AntonLapounov]

For my understanding: why is there a targetOS != null check?

targetOS is passed to the crossgen2 compiler to let it know what operating system to target and what JIT library to load. The only valid values recognized by the compiler are windows, linux, freebsd, and osx. Any other OS identifier must be mapped to one of these four values. If there is no existing API for this mapping, we may need to follow edges of the RID graph.

[AntonLapounov]

If yes, can we emulate the usual behavior?

Currently we are emulating the usual behavior because source-build builds the runtime repo twice.

I meant altering the SDK code so that the runtime identifier we get in a source-build is identical to what NuGetUtils.GetBestMatchingRid currently reports in normal builds. I assume a source-build uses the same RID graph, but a different availableRuntimeIdentifiers list, is that correct?

[tmds]

a different availableRuntimeIdentifiers list, is that correct?

Yes. When source-build is finished building, you get an SDK and a set of packages for its rid (like fedora.36-x64).

We'd like to use these non-portable rid packages. dotnet/runtime#75597 includes changes to make the SDK pick these up.

It sounds like we should extend targetOS logic so it looks at the runtime graph to map the non-portable rid to a value the crossgen2 compiler understands.

I think we may also run into cases where the rid is not (yet) in the runtime graph during source-build's runtime build, but we can see how to fix those if they occur.

[AntonLapounov]

It sounds like we should extend targetOS logic so it looks at the runtime graph to map the non-portable rid to a value the crossgen2 compiler understands.

How many different OS identifiers, like fedora, do we have to support for source-build? If the set is limited, we might just hard-code those IDs instead of using the graph.