Inconsistent `crossgen2` build failures

[dougbu]

Build Information

Build: https://runfo.azurewebsites.net/view/build/?number=1788535
Leg Name: *Windows*

Error Message

{ "ErrorMessage":"Unable to load native library 'Microsoft.DiaSymReader.Native'" }

If you hit this

The problem usually goes away on retry. Rerun failed jobs in the AzDO pipeline, Re-run in the Checks tab of your PR, add an /azp run comment in your PR (avoid this if you can, it's inefficient), or update your PR's branch.

Report

Summary

Day Hit Count	Week Hit Count	Month Hit Count
0	0	0

[dougbu]

@AlitzelMendez @missymessa @riarenas will the infrastructure handle wildcards in the Leg Name portion of the template❔ If not, what's the best way to handle both Build: Windows ARM and Build: Windows x64/x86❔

[dougbu]

Oops, just found a match in a Test: Windows Server x64 job 😦

[riarenas]

I think only the error message is important for the automation to get the issues picked up, and the leg Name is more so that we don't have to guess where exactly in a build with multiple jobs the error might have happened.

@AlitzelMendez and @ulisesh to confirm and/or correct my understanding.

[dougbu]

Thanks @riarenas

[ulisesh]

I think only the error message is important for the automation to get the issues picked up, and the leg Name is more so that we don't have to guess where exactly in a build with multiple jobs the error might have happened.

@AlitzelMendez and @ulisesh to confirm and/or correct my understanding.

That's correct. The leg name is for people who are investigating the issue but isn't used by the automation

[dougbu]

runfo query for this problem is https://runfo.azurewebsites.net/search/build-logs/?bq=started%3A~14%20definition%3Aaspnetcore-ci%20targetbranch%3Amain&lq=text%3A%22unable%20to%20load%20native%20library%20%27microsoft.diasymreader.native%27%22%20logname%3Awindows. Other recent failures (not all of which show up in this query right now) include:

• 20220518.35
• 20220519.24
• 20220521.3
• 20220521.4
• 20220523.4
• 20220523.6

[dougbu]

External issue is dotnet/runtime#69743. @MichalStrehovsky has dotnet/runtime#69842 open to get more information. Once that merges and we merge a dependency PR containing that work, the next failure here should help understand what's going wrong.

@BrennanConroy did you learn anything from the binary logs you were looking at yesterday❔

/fyi @MackinnonBuck if you're contacted about a failure that looks like this one, please check to make sure it's shown in the Build Analysis page under the Checks tab. I may have messed up the template.

[dougbu]

/cc @jkotas @am11

[dougbu]

Another question for @riarenas @ulisesh @AlitzelMendez @missymessa: I added a newline after the Error Message JSON because otherwise the closing triple-ticks weren't recognized and the If you hit this section was also considered JSON. Will this prevent associating failures w/ this issue❔

[am11]

symreader's README recommends:

It is recommended that new applications and libraries read Portable PDBs directly using APIs provided by System.Reflection.Metadata package. These APIs are much more efficient than DiaSymReader APIs. Microsoft.DiaSymReader.PortablePdb bridge is recommended for existings apps that already use DiaSymReader APIs and need to be able to read Portable PDBs without significant changes to their source.

System.Reflection.Metadata currently does not support reading Windows PDB (it only supports Portable PDB), I think that is why various components of runtime (including CG2) still depend on DiaSymReader.Native.

(If we want to continue to support Windows PDB in .NET indefinitely) Ideally, we can act on David's suggestion dotnet/runtime#43904 (comment) and add managed WindowsPdbReader/Writer to get rid of symreader dependency once in for all, but it is a non-trivial task. https://github.com/microsoft/microsoft-pdb and LLVM's PDB reader and writer are likely the most relevant / battle-tested references, if someone is looking to write managed counterpart.

[agocke]

FYI @MichalStrehovsky, I'm not sure if this could be a problem with native resource loading in NativeAOT. I downloaded the package and it has the following files:

clrjit_universal_arm_x64.dll
clrjit_win_x64_x64.dll crossgen2.exe
Microsoft.DiaSymReader.Native.amd64.dll
clrjit_universal_arm64_x64.dll
clrjit_win_x86_x64.dll
crossgen2.runtimeconfig.json
clrjit_unix_x64_x64.dll
crossgen2.deps.json
jitinterface_x64.dll

[ulisesh]

Another question for @riarenas @ulisesh @AlitzelMendez @missymessa: I added a newline after the Error Message JSON because otherwise the closing triple-ticks weren't recognized and the If you hit this section was also considered JSON. Will this prevent associating failures w/ this issue❔

It shouldn't prevent it, the way it is written right now should work