Explain stages in terms of the compiler currently running

Author: jyn514

src/building/bootstrapping.md

@@ -57,6 +57,8 @@ first build the new compiler with an older compiler and then use that to
 build the new compiler with itself. For development, you usually only want
 the `stage1` compiler: `x.py build library/std`.
 
+For more information about stages, see <#understanding-stages-of-bootstrap> below.
+
 ## Complications of bootstrapping
 
 Since the build system uses the current beta compiler to build the stage-1
@@ -108,50 +110,89 @@ contribution [here][bootstrap-build].
 
 ## Understanding stages of bootstrap
 
-This is a detailed look into the separate bootstrap stages. When running
-`x.py` you will see output such as:
-
-```txt
-Building stage0 std artifacts
-Copying stage0 std from stage0
-Building stage0 compiler artifacts
-Copying stage0 rustc from stage0
-Building LLVM for x86_64-apple-darwin
-Building stage0 codegen artifacts
-Assembling stage1 compiler
-Building stage1 std artifacts
-Copying stage1 std from stage1
-Building stage1 compiler artifacts
-Copying stage1 rustc from stage1
-Building stage1 codegen artifacts
-Assembling stage2 compiler
-Uplifting stage1 std
-Copying stage2 std from stage1
-Generating unstable book md files
-Building stage0 tool unstable-book-gen
-Building stage0 tool rustbook
-Documenting standalone
-Building rustdoc for stage2
-Documenting book redirect pages
-Documenting stage2 std
-Building rustdoc for stage1
-Documenting stage2 whitelisted compiler
-Documenting stage2 compiler
-Documenting stage2 rustdoc
-Documenting error index
-Uplifting stage1 rustc
-Copying stage2 rustc from stage1
-Building stage2 tool error_index_generator
-```
+### Overview
+
+This is a detailed look into the separate bootstrap stages.
+
+The convention `x.py` uses is that:
+- A `--stage N` flag means to run the stage N compiler (`stageN/rustc`).
+- A "stage N artifact" is an artifact that is _produced_ by the stage N compiler.
+- The "stage (N+1) compiler" is assembled from "stage N artifacts". This process is called _uplifting_.
+
+For example, `x.py build --stage 0` means to build with the beta `rustc`, and
+`test --stage 0` isn't really meaningful (it's not running tests on your changes,
+but on `beta`, so it always fails). Similarly, `doc --stage 0` means to
+document using the beta `rustdoc`.
+
+Note that this implies the stage N compiler is _not_ the same as the compiler
+built by `build --stage N compiler/rustc` -- that's 'stage N artifacts'
+('the compiler built by stage N').
+
+In short, _stage 0 uses the stage0 compiler to create stage0 artifacts which
+will later be uplifted to stage1_.
+
+In each stage, two major steps are performed:
+
+1. `std` is compiled by the stage N compiler.
+2. That `std` is linked to programs built by the stage N compiler, including the stage (N+1) compiler.
+
+This is somewhat intuitive if one thinks of the stage (N+1) compiler as "just"
+another program we are building with the stage N compiler:
+`build --stage N compiler/rustc` is linking the stage (N+1) compiler to the `std`
+built by the stage N compiler.
 
-A deeper look into `x.py`'s phases can be seen here:
+Here is a chart of a full build using x.py:
 
 <img alt="A diagram of the rustc compilation phases" src="../img/rustc_stages.svg" class="center" />
 
 Keep in mind this diagram is a simplification, i.e. `rustdoc` can be built at
 different stages, the process is a bit different when passing flags such as
 `--keep-stage`, or if there are non-host targets.
 
+The stage 1 artifacts are what is shipped to end-users, including `stage2/bin/rustc`.
+
+### Stages and libstd
+
+Note that there are two `std` libraries in play here:
+1. The library _linked_ to `stageN/rustc`, which was built by stage N-1 (stage N-1 `std`)
+2. The library _used to compile programs_ with `stageN/rustc`, which was built by stage N (stage N `std`).
+
+stage N `std` is pretty much necessary for any useful work with the compiler.
+Without it, you can only compile programs with `#![no_core]` -- not terribly useful!
+
+The reason these need to be different is because they aren't necessarily ABI-compatible:
+there could be a new layout optimization on nightly that isn't present in `beta`.
+
+This is also where `--keep-stage 1 library/std` comes into play. Since most
+changes to the compiler don't actually change the ABI, once you've produced a
+`std` in stage 1, you can probably just reuse it with a different compiler.
+If the ABI hasn't changed, you're good to go, no need to spend the time
+recompiling that `std`.
+`--keep-stage` simply assumes the previous compile is fine and copies those
+artifacts into the appropriate place, skipping the cargo invocation.
+
+### Cross-compiling
+
+Building stage2 `std` is different depending on whether you are cross-compiling or not
+(see in the table how stage2 only builds non-host `std` targets).
+This is because `x.py` uses a trick: if `HOST` and `TARGET` are the same,
+it will reuse stage1 `std` for stage2! This is sound because stage1 `std`
+was compiled with the stage1 compiler, i.e. a compiler using the source code
+you currently have checked out. So it should be identical (and therefore ABI-compatible)
+to the `std` that `stage2/rustc` would compile.
+
+However, when cross-compiling, stage1 `std` will only run on the host.
+So the stage2 compiler has to recompile `std` for the target.
+
+### Why does only libstd use cfg(bootstrap)?
+
+The `rustc` generated by the stage0 compiler is linked to the freshly-built
+`std`, which means that for the most part only `std` needs to be cfg-gated,
+so that `rustc` can use features added to std immediately after their addition,
+without need for them to get into the downloaded beta.
+
+### Directories and artifacts generated by x.py
+
 The following tables indicate the outputs of various stage actions:
 
 | Stage 0 Action                                            | Output                                       |
@@ -164,7 +205,7 @@ The following tables indicate the outputs of various stage actions:
 | copy `stage0-rustc (except executable)`                   | `build/HOST/stage0-sysroot/lib/rustlib/HOST` |
 | build `llvm`                                              | `build/HOST/llvm`                            |
 | `stage0` builds `codegen` with `stage0-sysroot`           | `build/HOST/stage0-codegen/HOST`             |
-| `stage0` builds `rustdoc` with `stage0-sysroot`           | `build/HOST/stage0-tools/HOST`               |
+| `stage0` builds `rustdoc`, `clippy`, `miri`, with `stage0-sysroot` | `build/HOST/stage0-tools/HOST`      |
 
 `--stage=0` stops here.
 
@@ -192,80 +233,6 @@ The following tables indicate the outputs of various stage actions:
 
 `--stage=2` stops here.
 
-Note that the convention `x.py` uses is that:
-- A "stage N artifact" is an artifact that is _produced_ by the stage N compiler.
-- The "stage (N+1) compiler" is assembled from "stage N artifacts".
-- A `--stage N` flag means build _with_ stage N.
-
-In short, _stage 0 uses the stage0 compiler to create stage0 artifacts which
-will later be uplifted to stage1_.
-
-Every time any of the main artifacts (`std` and `rustc`) are compiled, two
-steps are performed.
-When `std` is compiled by a stage N compiler, that `std` will be linked to
-programs built by the stage N compiler (including `rustc` built later
-on). It will also be used by the stage (N+1) compiler to link against itself.
-This is somewhat intuitive if one thinks of the stage (N+1) compiler as "just"
-another program we are building with the stage N compiler. In some ways, `rustc`
-(the binary, not the `rustbuild` step) could be thought of as one of the few
-`no_core` binaries out there.
-
-So "stage0 std artifacts" are in fact the output of the downloaded stage0
-compiler, and are going to be used for anything built by the stage0 compiler:
-e.g. `rustc` artifacts. When it announces that it is "building stage1
-std artifacts" it has moved on to the next bootstrapping phase. This pattern
-continues in latter stages.
-
-Also note that building host `std` and target `std` are different based on the
-stage (e.g. see in the table how stage2 only builds non-host `std` targets.
-This is because during stage2, the host `std` is uplifted from the "stage 1"
-`std` -- specifically, when "Building stage 1 artifacts" is announced, it is
-later copied into stage2 as well (both the compiler's `libdir` and the
-`sysroot`).
-
-This `std` is pretty much necessary for any useful work with the compiler.
-Specifically, it's used as the `std` for programs compiled by the newly compiled
-compiler (so when you compile `fn main() { }` it is linked to the last `std`
-compiled with `x.py build library/std`).
-
-The `rustc` generated by the stage0 compiler is linked to the freshly-built
-`std`, which means that for the most part only `std` needs to be cfg-gated,
-so that `rustc` can use featured added to std immediately after their addition,
-without need for them to get into the downloaded beta. The `std` built by the
-`stage1/bin/rustc` compiler, also known as "stage1 std artifacts", is not
-necessarily ABI-compatible with that compiler.
-That is, the `rustc` binary most likely could not use this `std` itself.
-It is however ABI-compatible with any programs that the `stage1/bin/rustc`
-binary builds (including itself), so in that sense they're paired.
-
-This is also where `--keep-stage 1 library/std` comes into play. Since most
-changes to the compiler don't actually change the ABI, once you've produced a
-`std` in stage 1, you can probably just reuse it with a different compiler.
-If the ABI hasn't changed, you're good to go, no need to spend the time
-recompiling that `std`.
-`--keep-stage` simply assumes the previous compile is fine and copies those
-artifacts into the appropriate place, skipping the cargo invocation.
-
-The reason we first build `std`, then `rustc`, is largely just
-because we want to minimize `cfg(stage0)` in the code for `rustc`.
-Currently `rustc` is always linked against a "new" `std` so it doesn't
-ever need to be concerned with differences in std; it can assume that the std is
-as fresh as possible.
-
-The reason we need to build it twice is because of ABI compatibility.
-The beta compiler has it's own ABI, and then the `stage1/bin/rustc` compiler
-will produce programs/libraries with the new ABI.
-We used to build three times, but because we assume that the ABI is constant
-within a codebase, we presume that the libraries produced by the "stage2"
-compiler (produced by the `stage1/bin/rustc` compiler) is ABI-compatible with
-the `stage1/bin/rustc` compiler's produced libraries.
-What this means is that we can skip that final compilation -- and simply use the
-same libraries as the `stage2/bin/rustc` compiler uses itself for programs it
-links against.
-
-This `stage2/bin/rustc` compiler is shipped to end-users, along with the
-`stage 1 {std,rustc}` artifacts.
-
 ## Passing stage-specific flags to `rustc`
 
 `x.py` allows you to pass stage-specific flags to `rustc` when bootstrapping.