Strip duplicate files from SharedFx input to Nuget Pack

[wtgodbe]

Should resolve #25293

Sometimes, the call to ResolveReferences in the SharedFx project will add .pdb's to ReferenceCopyLocalPaths that are in the output directory of individual projects. This means we add the same .pdb twice (w/ 2 different paths) to the input of Nuget Pack, causing nuget errors. We're still investigating why this behavior exists, but we should also be defensive about duplicate files and strip them out.

@Pilchie can this go in as tell-mode for rc2?

[wtgodbe]

A couple internal builds to get extra validation that this transient issue is fixed:

https://dev.azure.com/dnceng/internal/_build/results?buildId=807393&view=results
https://dev.azure.com/dnceng/internal/_build/results?buildId=807397&view=results

[Pilchie]

Does the Distinct keep the "right" version of the PDB? Does it matter, are they identical in the two locations?

[wtgodbe]

Does the Distinct keep the "right" version of the PDB? Does it matter, are they identical in the two locations?

I think distinct just picks whichever one is "first" in the list - but, the .pdb's are the same in both locations (the sharedFx project copies them over from their original locations)

[dougbu]

Wondering about the tradeoffs here. Is this a better option than disabling the RAR cache❔

[wtgodbe]

Wondering about the tradeoffs here. Is this a better option than disabling the RAR cache❔

I'll follow up with msbuild folks about this. Also have some test builds going for that scenario:

https://dev.azure.com/dnceng/internal/_build/results?buildId=807440&view=results
https://dev.azure.com/dnceng/internal/_build/results?buildId=807444&view=results

[halter73]

Awesome! This does feel like it shouldn't be necessary, but I'll take anything that deflakes the build at this point.

[Pilchie]

I'm fine with either this or the disable RAR cache fix through tell mode once you settle with MSBuild folks on what is the better approach.

[wtgodbe]

One of the test builds for disabling the RAR cache saw the error recur, while the other didn't - all three builds of this branch have succeeded. I'm inclined to go ahead with this change, but will wait to hear back from the msbuild folks before merging.