Add root build definition which builds the entire repo

[ViktorHofer]

Ported from @jashook's doc: https://github.com/dotnet/consolidation/pull/30

Overview

The purpose of this comment is to give a design of the CI build of dontet/runtime using the live builds of each folder under src. This replace the old package restore dependency flow between what used to be dotnet/coreclr, dotnet/corefx and dotnet/core-setup.

It is important to note that this focuses on the CI build of the dotnet/runtime product. This distinction is required because the developer worklow for the repository is different and is being added here: #55.

Goal

We want to have a single pipeline that builds a minimum matrix of coreclr, libraries and installer for all paths in the repository excluding docs/*.

The simple justification is that our setup already mostly requires this. Coreclr changes have to build libraries to run tests. Libraries has to build CoreCLR to build and to run tests once we're live / live. The only real delta here is whether or not to build installers. That can be done in parallel with testing though hence it doesn't affect throughput.

The approach would be to condition a full platform matrix and condition test runs based on changed paths. There would be an early job which defines variables (i.e changedLibraries, changedCoreClr) and then we would condition build and test jobs based on those variables.

The downside to this approach is that a bunch of jobs will be shown as skipped when the path doesn't applies, but long-term we can request a feature that defines a variable at compile time with the changed paths and then we can condition at compile time rather than run time.

The advantages is that now we have less .yml entry-points to maintain and we avoid a lot of build duplication on our CI when a change touches multiple subsets.

Building

runtime/src/coreclr and runtime/src/corefx should each have templates which allow the following parameters, platforms: [<OS>_<Arch>], buildConfig [debug, checked, release], jobTemplate.

Sample yml:

- template: templates/platform-matrix.yml
  parameters:
    jobTemplate: coreclr/build-job.yml
    buildConfig: debug
    platforms:
    - Windows_NT_x64
    - Windows_NT_x86

    # Job parameters is a grab bag of extra properties.
    jobParameters:
      testGroup: innerloop

This will allow us to re-use the platform fan out template logic for the coreclr and libraries steps build.

This is about to be done for coreclr and libraries (#294 and #274).

Pipeline mock up

1. Official Build:
   build coreclr -> build libraries -> build installer -> sign -> publish
2. changedCoreClr == true:
   build coreclr (full matrix including checked builds) -> build libraries -> build libraries tests, build coreclr managed tests, build coreclr native tests -> run tests
3. changedLibraries == true:
   build coreclr -> build libraries -> build libraries tests -> run libraries tests
4. changedInstaller == true:
   build coreclr -> build libraries -> build installer -> build installer tests -> run installer tests

cc @dotnet/runtime-infrastructure

[trylek]

Adding @jkoritzinsky who is currently making progress on "live live build" as there seems to be quite a bit of overlap to prevent duplication of efforts.

[safern]

Assigning to me as I already started with this work.

Just wanted to make sure we're all in the same page as we've had super long discussions on Friday on what the plan is for this.  (You guys were OOF).

@jashook @sbomer and I, discussed on the structure for this and we think we shouldn't put an effort to put up a pipeline that builds the entire repo just yet as we should try and re-use as much as possible from the CI builds in order to parallelyze. Also @dagood was going to put a pipeline that uses the local live live logic that @jkoritzinsky put up together for one configuration in order to start the signing work and then the signing work should just be ported over to the "libraries" live live CI build as part of the official build.

As a result of that @jashook put up together this file: https://github.com/dotnet/consolidation/pull/30

[ViktorHofer]

Thanks for summarizing the status. Removing my assignment as work is already in progress here.

[safern]

I've updated the description to summarize the last plan that we discussed today in the morning.

Please feel free to update and comment with thoughts and opinions if you agree or disagree with something on that plan.

[trylek]

Sounds great to me. As to the nearest sequencing, my expectations are the following:

1. I'll rename the live-live pipeline from eng/pipelines/coreclr/pr.yml to a new name (my current plan is eng/pipelines/root.yml but it's certainly up to further discussion) and I'll put back the pre-existing CoreCLR pr.yml - this is also important to verify that my changes haven't broken the "legacy" runtime-coreclr pipeline;
2. I'll make my best to address the rest of PR feedback and merge the change in ASAP, ideally before EOD today;
3. Once the change is in, @dagood can start working on wiring up the installer logic;
4. In parallel, we can work on finalizing the test coverage extent and other details regarding switch-over of innerloop testing in the individual subrepos and an overall outerloop testing of the entire runtime repo to the live-live logic.

[ViktorHofer]

and an overall outerloop testing of the entire runtime repo

Hearing of that the first time. Do we actually want that?