Rethink Binary Scanner Implementation

[ellahathaway]

We need to decide on some changes to the binary scanner as part of the work being done in #4088:

• Move binary-scanner outside of arcade-services: The binary scanner is currently part of the arcade-services repository. Since we're now allowing binaries in the VMR, it would likely serve better as a smoke test for source-build only. See Remove binary cloaks from the VMR's main branch #4087 (comment).

• Run binary-scanner separate from scanning: Currently, the binary-scanner runs as part of the tag-and-scan pipeline stage. As part of Provide a script to remove all checked binaries from your local environment #4089, binaries are removed from the VMR when source-building as part of ./prep.sh which runs after tag-and-scan. Thus, binary scanning will have to take place after ./prep.sh when source-building.

cc @MichaelSimons @premun @dkurepa @dotnet/source-build-internal

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ellahathaway]

Related to #4108

[dkurepa]

Hi Ella, reading the issue description it looks like we should make a change to our pipeline yamls, not move the binary scanner out of arcade-services, or am I missing something? The VMR scanner is currently implemented as a darc command, and the only things it needs is the vmr path, and the baseline file, so it should be quite flexible

[MichaelSimons]

@dkurepa - The motivation behind the first item is that distro maintainers will be stripping the binaries themselves via a script we provide. It would be ideal if we also provided a way for them to self-validate the repo is binary clean. That implies the validation logic needs to reside within the VMR. Using darc is a non-starter for this usage scenario.

[premun]

The scan itself is just running diff --numstat over the right folders with the right pathspecs (exclusions) and parsing its output.
The only logic worth any consideration is parsing the source-mappings file to get the list of exclusions (or inclusions) and parsing the source-manifest.json file to understand which directories are inlined submodules. Since this will be simplified a lot, it might be okay to just move/copy the code out in some limited form. But I will leave this up to you.

I understand we will need to also use the same logic in the binary deletion script/tool so I'd still probably advise to keep it in C#.
A bash version is still border-line possible too I guess if that's easier to make it part of prep.sh more natively.

Alternatively, we could possibly move the source mappings parser into a library in Arcade but this will complicate our lives considerably (arcade-services is using 8.0 Arcade etc). So I don't think it's worth the hassle just for a piece of possibly duplicated code for parsing a simple JSON file.

[ellahathaway]

Based on dotnet/installer#18563 (comment), the source-mappings file will only be used for permanent exclusions that both MSFT and distro maintainers don't want (licenses). With this new implementation, the binary scanner will not have to be modified to use source-mappings. However, we are changing how we detect and remove binaries. That is, each repo will have a separate list of files/patterns that should be deleted and validated against with infra that we will build up in Arcade. This infra will be keyed off in SB repo legs, similarly to how prebuilt detection runs. As such, the binary-scanner no longer serves a purpose in the VMR. Instead, we will have to build up infra in arcade, and transition from the binary-scanner to new tooling in Arcade.