bootstrap: Add snapshot tests for path-to-step handling

[Zalathar]

This PR adds a suite of snapshot tests for how bootstrap translates command-line “paths” (which are not necessarily actual paths) into a set of top-level steps to execute.

Unlike the existing suite of snapshot tests (for transitive step logging), I decided to use .snap files over inline snapshots, because I find that inline snapshots quickly make test files impossible to navigate. Instead, I set up a macro that makes it relatively easy to add or modify test cases.

Using .snap files also means that the tests can be blessed by setting INSTA_UPDATE=always, without necessarily needing the cargo insta tool installed, though the tool can still be used instead if desired.

These snapshot tests capture current behavior, to prevent unintended changes or regressions. If the current behavior is wrong or undersirable, then any fix will necessarily have to re-bless the affected tests!

r? Kobzol

[Zalathar]

It’s worrying that CI sees different snapshots.

I wonder if the “real” host/target is bleeding through somehow?

[Zalathar]

Oh gross, x test includes bootstrap only when a CI environment is detected.

Still not sure what's going on with the rustdoc JS tests; perhaps something is detecting that I don't have the requisite paths set on my machine?

[Zalathar]

Yeah, looks like those steps auto-skip themselves when build.nodejs is not set and fails to resolve to an ambient tool in the current PATH.

[Kobzol]

Thanks for working on this! This looks great. I wonder if we should just use out-of-band snap files also for the "normal" snapshot tests? It seems a bit weird to have two systems (in-band and out-of-band) in bootstrap.

Yeah, looks like those steps auto-skip themselves when build.nodejs is not set and fails to resolve to an ambient tool in the current PATH.

That's a problem, right? Because if someone does/doesn't have node available might make those tests to fail locally. That might be annoying especially when they have to be re-blessed locally.

[Zalathar]

Right, that’s why the test overwrites build.config.nodejs after creating the builder, to pretend that node was successfully discovered even if it wasn’t actually found on the system that’s running the tests.

(E.g. I currently don’t have node on my path, which is why I had to add that override, so that my machine would generate the same snapshots as a CI machine that does have node installed.)

[Kobzol]

Sorry, I didn't realize that you overwrite it after parsing the config, that makes sense.

The paths will probably have to be blessed from time to time when e.g. rustc crates change, but that's not that common, I suppose.

In general, I like this approach. You can r=me, unless you want to e.g. add more tests in this PR, or something.

[Zalathar]

Adding more tests later is pretty easy, so I'm happy to get the plumbing and current tests merged for now.

@bors r=Kobzol

[bors]

📌 Commit bb05869 has been approved by Kobzol

It is now in the queue for this repository.

[jieyouxu]

Very cool 👍

[Zalathar]

Failed in rollup: #148845 (comment)

@bors r-

[rustbot]

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

[Zalathar]

I added some quick-and-dirty backslash normalization to avoid failures on Windows, and also added a handful of extra tests related to library.

@bors try jobs=x86_64-msvc-2,test-various

[rust-bors]

☀️ Try build successful (CI)
Build commit: 3c7eea9 (3c7eea994e6ee0660391247b643d5db65e88fc76, parent: 11339a0ef5ed586bb7ea4f85a9b7287880caac3a)

[Zalathar]

@rustbot ready

[Kobzol]

@bors r+

[bors]

📌 Commit 2412e15 has been approved by Kobzol

It is now in the queue for this repository.

[bors]

⌛ Testing commit 2412e15 with merge 95aeb46...

[bors]

☀️ Test successful - checks-actions
Approved by: Kobzol
Pushing 95aeb46 to main...

[github-actions]

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 0b329f8 (parent) -> 95aeb46 (this PR)

Test differences

Show 39 test diffs

Stage 0

• core::builder::cli_paths::tests::no_unused_snapshots: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_build: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_build_compiler: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_build_compiletest: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_build_library: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_build_llvm: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_build_rustc: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_build_rustc_llvm: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_build_rustdoc: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_build_sysroot: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_check: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_check_bootstrap: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_check_compiler: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_check_compiletest: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_check_compiletest_include_default_paths: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_check_library: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_check_rustc: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_check_rustdoc: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_dist: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_test: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_test_coverage: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_test_coverage_map: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_test_coverage_run: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_test_coverage_skip_coverage_run: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_test_debuginfo: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_test_library: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_test_librustdoc: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_test_librustdoc_rustdoc: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_test_rustdoc: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_test_skip_coverage: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_test_skip_tests: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_test_skip_tests_etc: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_test_tests: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_test_tests_skip_coverage: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_test_tests_ui: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_test_tidy: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_test_tidyselftest: [missing] -> pass (J0)
• core::builder::cli_paths::tests::x_test_ui: [missing] -> pass (J0)

Additionally, 1 doctest diff were found. These are ignored, as they are noisy.

Job group index

• J0: aarch64-apple, aarch64-gnu, aarch64-gnu-llvm-20-2, aarch64-msvc-2, i686-gnu-2, i686-gnu-nopt-2, i686-msvc-2, x86_64-gnu, x86_64-gnu-llvm-20-1, x86_64-gnu-llvm-20-3, x86_64-gnu-llvm-21-1, x86_64-gnu-llvm-21-3, x86_64-gnu-nopt, x86_64-gnu-stable, x86_64-mingw-2, x86_64-msvc-2

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
    test-dashboard 95aeb4696545eb4c9cbb68516f2912770e3846ea --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

1. aarch64-apple: 9312.2s -> 6779.1s (-27.2%)
2. pr-check-1: 1975.3s -> 1499.8s (-24.1%)
3. x86_64-gnu-aux: 6977.9s -> 5756.6s (-17.5%)
4. x86_64-gnu-llvm-21-2: 6155.0s -> 5139.3s (-16.5%)
5. x86_64-gnu-tools: 3923.1s -> 3341.2s (-14.8%)
6. x86_64-rust-for-linux: 3094.9s -> 2658.8s (-14.1%)
7. x86_64-gnu-llvm-20: 2899.6s -> 2492.7s (-14.0%)
8. x86_64-gnu-llvm-21-1: 3747.7s -> 3270.4s (-12.7%)
9. i686-gnu-2: 6324.2s -> 5520.6s (-12.7%)
10. x86_64-gnu-miri: 5227.5s -> 4576.1s (-12.5%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.