Create and publish a MergedManifest for unified builds

[dkurepa]

After we're done with #4198, we'll have a VerticalManifest published for each leg. We'll need to use all of these manifests to produce a MergedManifest, similarly to how we do it currently in individual repos.
We'll need to have some custom logic for this, as multiple legs might produce duplicate assets #4157.

[dkurepa]

Based on the discussions we had offline, we will implement a task that will be used in two scenarios:

• In the join point builds, it will prepare the machine for builds, by downloading necessary assets
• In the final join point it will create a merged manifest and download all assets, preparing for artifact publishing (BlobArtifacts and PackageArtifacts)

Based on these requirements, the task will do the following

• The input will be the name of the "main" vertical manifest, and the list of all vertical manifests
• The task will create a merged manifest by taking all artifacts from the main manifest, and then going through all other manifests and adding artifacts that are not there already
• As it's going through these manifests, it will download pipeline artifacts that corelate to the entries it added to the manifest (it will always download the full main vertical, and then, for example, it takes Microsoft.NETCore.App.Crossgen2.linux-x64 from the Ubuntu2204_x64 vertical`)
• The final result of this task should be a MergedManifest, and build artifacts that correlate to the entries in it. These artifacts will be used for the join point builds, or we'll publish them as BlobArtifacts and PackageArtifacts which are special artifacts that are used by the Maestro publishing pipeline. The MergedManifest will only be used in the final join point, to produce a final MergedManifest. This manifest will be passed to the PushMetadataToBuildAssetRegistry task to create a bar build

[dkurepa]

For downloading pipeline artifacts, we have two options:

• Figure out which artifacts we want to download for each leg, and download them one by one
• Download whole artifact legs, copy what we need, and delete the rest

The main advantage with the first approach is that we'd be downloading less amount of data, we'd be taking just what we need. But this approach also has a fatal flaw, we'd download every file individually so, we'd probably exceed our rate limit, which would cause throttling. Because of this we probably shouldn't use this approach.

The other approach flaw is that we might run out of space, since it requires downloading an artifact zip, and then unzipping it. We'd just save the packages we need, deleting the reset, and could optimize around download order to minimize this risk

[mmitche]

The main advantage with the first approach is that we'd be downloading less amount of data, we'd be taking just what we need. But this approach also has a fatal flaw, we'd download every file individually so, we'd probably exceed our rate limit, which would cause throttling. Because of this we probably shouldn't use this approach.

We don't hit this rate limit when doing our BAR publishing today, so it's quite possible we wouldn't hit it.