Merge branch 'master' into tempfile

Author: Jefffrey

.github/workflows/ci.yml

@@ -10,17 +10,14 @@ on:
     branches:
       - 'master'
   schedule:
-    - cron: '6 6 * * *' # At 6:06 UTC every day.
+    - cron: '44 4 * * *' # At 4:44 UTC every day.
 
-env:
-  CARGO_UNSTABLE_SPARSE_REGISTRY: 'true'
+defaults:
+  run:
+    shell: bash
 
 jobs:
   build:
-    runs-on: ${{ matrix.os }}
-    env:
-      RUST_BACKTRACE: 1
-      HOST_TARGET: ${{ matrix.host_target }}
     strategy:
       fail-fast: false
       matrix:
@@ -31,13 +28,13 @@ jobs:
             host_target: x86_64-apple-darwin
           - os: windows-latest
             host_target: i686-pc-windows-msvc
+    runs-on: ${{ matrix.os }}
+    env:
+      RUST_BACKTRACE: 1
+      HOST_TARGET: ${{ matrix.host_target }}
     steps:
       - uses: actions/checkout@v3
 
-      - name: Set the tag GC interval to 1 on linux
-        if: runner.os == 'Linux'
-        run: echo "MIRIFLAGS=-Zmiri-tag-gc=1" >> $GITHUB_ENV
-
       # Cache the global cargo directory, but NOT the local `target` directory which
       # we cannot reuse anyway when the nightly changes (and it grows quite large
       # over time).
@@ -54,17 +51,14 @@ jobs:
             # contains package information of crates installed via `cargo install`.
             ~/.cargo/.crates.toml
             ~/.cargo/.crates2.json
-          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
-          restore-keys: ${{ runner.os }}-cargo
+          key: ${{ runner.os }}-cargo-reset20230315-${{ hashFiles('**/Cargo.lock') }}
+          restore-keys: ${{ runner.os }}-cargo-reset20230315
 
       - name: Install rustup-toolchain-install-master
         if: ${{ steps.cache.outputs.cache-hit != 'true' }}
-        shell: bash
-        run: |
-          cargo install -f rustup-toolchain-install-master
+        run: cargo install -f rustup-toolchain-install-master
 
       - name: Install "master" toolchain
-        shell: bash
         run: |
           if [[ ${{ github.event_name }} == 'schedule' ]]; then
             echo "Building against latest rustc git version"
@@ -79,7 +73,7 @@ jobs:
           cargo -V
 
       - name: Test
-        run: bash ./ci.sh
+        run: ./ci.sh
 
   style:
     name: style checks
@@ -106,18 +100,19 @@ jobs:
             # contains package information of crates installed via `cargo install`.
             ~/.cargo/.crates.toml
             ~/.cargo/.crates2.json
-          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
-          restore-keys: ${{ runner.os }}-cargo
+          key: ${{ runner.os }}-cargo-reset20230315-${{ hashFiles('**/Cargo.lock') }}
+          restore-keys: ${{ runner.os }}-cargo-reset20230315
 
       - name: Install rustup-toolchain-install-master
         if: ${{ steps.cache.outputs.cache-hit != 'true' }}
-        shell: bash
-        run: |
-          cargo install -f rustup-toolchain-install-master
+        run: cargo install -f rustup-toolchain-install-master
 
       - name: Install "master" toolchain
-        shell: bash
         run: |
+          if [[ ${{ github.event_name }} == 'schedule' ]]; then
+            echo "Building against latest rustc git version"
+            git ls-remote https://github.com/rust-lang/rust/ HEAD | cut -f 1 > rust-version
+          fi
           ./miri toolchain
 
       - name: Show Rust version
@@ -131,14 +126,13 @@ jobs:
       - name: clippy
         run: ./miri clippy -- -D warnings
       - name: rustdoc
-        run: RUSTDOCFLAGS="-Dwarnings" cargo doc --document-private-items
+        run: RUSTDOCFLAGS="-Dwarnings" ./miri cargo doc --document-private-items
 
   # These jobs doesn't actually test anything, but they're only used to tell
   # bors the build completed, as there is no practical way to detect when a
   # workflow is successful listening to webhooks only.
   #
   # ALL THE PREVIOUS JOBS NEED TO BE ADDED TO THE `needs` SECTION OF THIS JOB!
-  # (`fmt` is deliberately not listed, we want bors to ignore it.)
   end-success:
     name: bors build finished
     runs-on: ubuntu-latest
@@ -156,27 +150,65 @@ jobs:
       - name: mark the job as a failure
         run: exit 1
 
-  # Send a Zulip notification when a cron job fails
   cron-fail-notify:
     name: cronjob failure notification
     runs-on: ubuntu-latest
     needs: [build, style]
     if: github.event_name == 'schedule' && (failure() || cancelled())
     steps:
+      # Send a Zulip notification
       - name: Install zulip-send
         run: pip3 install zulip
       - name: Send Zulip notification
-        shell: bash
         env:
           ZULIP_BOT_EMAIL: ${{ secrets.ZULIP_BOT_EMAIL }}
           ZULIP_API_TOKEN: ${{ secrets.ZULIP_API_TOKEN }}
         run: |
-          ~/.local/bin/zulip-send --stream miri --subject "Cron Job Failure (miri, $(date -u +%Y-%m))" \
+          ~/.local/bin/zulip-send --user $ZULIP_BOT_EMAIL --api-key $ZULIP_API_TOKEN --site https://rust-lang.zulipchat.com \
+            --stream miri --subject "Cron Job Failure (miri, $(date -u +%Y-%m))" \
             --message 'Dear @*T-miri*,
 
-          It would appear that the Miri cron job build failed. Would you mind investigating this issue?
+          It would appear that the [Miri cron job build]('"https://github.com/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"') failed.
+          
+          This likely means that rustc changed the miri directory and
+          we now need to do a [`./miri rustc-pull`](https://github.com/rust-lang/miri/blob/master/CONTRIBUTING.md#importing-changes-from-the-rustc-repo).
+
+          Would you mind investigating this issue?
 
           Thanks in advance!
           Sincerely,
-          The Miri Cronjobs Bot' \
-            --user $ZULIP_BOT_EMAIL --api-key $ZULIP_API_TOKEN --site https://rust-lang.zulipchat.com
+          The Miri Cronjobs Bot'
+
+      # Attempt to auto-sync with rustc
+      - uses: actions/checkout@v3
+        with:
+          fetch-depth: 256 # get a bit more of the history
+      - name: install josh-proxy
+        run: RUSTFLAGS="--cap-lints warn" cargo +stable install josh-proxy --git https://github.com/josh-project/josh --tag r22.12.06
+      - name: setup bot git name and email
+        run: |
+          git config --global user.name 'The Miri Conjob Bot'
+          git config --global user.email '[email protected]'
+      - name: get changes from rustc
+        run: ./miri rustc-pull
+      - name: Install rustup-toolchain-install-master
+        run: cargo install -f rustup-toolchain-install-master
+      - name: format changes (if any)
+        run: |
+          ./miri toolchain
+          ./miri fmt --check || (./miri fmt && git commit -am "fmt")
+      - name: Push changes to a branch
+        run: |
+          BRANCH="rustup-$(date -u +%Y-%m-%d)"
+          git switch -c $BRANCH
+          git push -u origin $BRANCH
+      - name: Create Pull Request
+        run: |
+          PR=$(gh pr create -B master --title 'Automatic Rustup' --body '')
+          ~/.local/bin/zulip-send --user $ZULIP_BOT_EMAIL --api-key $ZULIP_API_TOKEN --site https://rust-lang.zulipchat.com \
+            --stream miri --subject "Cron Job Failure (miri, $(date -u +%Y-%m))" \
+            --message "A PR doing a rustc-pull [has been automatically created]($PR) for your convenience."
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          ZULIP_BOT_EMAIL: ${{ secrets.ZULIP_BOT_EMAIL }}
+          ZULIP_API_TOKEN: ${{ secrets.ZULIP_API_TOKEN }}

CONTRIBUTING.md

@@ -71,19 +71,12 @@ and you can (cross-)run the entire test suite using:
 MIRI_TEST_TARGET=i686-unknown-linux-gnu ./miri test
 ```
 
-If your target doesn't support libstd, you can run miri with
+If your target doesn't support libstd that should usually just work. However, if you are using a
+custom target file, you might have to set `MIRI_NO_STD=1`.
 
-```
-MIRI_NO_STD=1 MIRI_TEST_TARGET=thumbv7em-none-eabihf ./miri test tests/fail/alloc/no_global_allocator.rs
-MIRI_NO_STD=1 ./miri run tests/pass/no_std.rs --target thumbv7em-none-eabihf
-```
-
-to avoid attempting (and failing) to build libstd. Note that almost no tests will pass
-this way, but you can run individual tests.
-
-`./miri test FILTER` only runs those tests that contain `FILTER` in their
-filename (including the base directory, e.g. `./miri test fail` will run all
-compile-fail tests).
+`./miri test FILTER` only runs those tests that contain `FILTER` in their filename (including the
+base directory, e.g. `./miri test fail` will run all compile-fail tests). These filters are passed
+to `cargo test`, so for multiple filers you need to use `./miri test -- FILTER1 FILTER2`.
 
 You can get a trace of which MIR statements are being executed by setting the
 `MIRI_LOG` environment variable.  For example:
@@ -107,7 +100,7 @@ evaluation error was originally raised.
 ### UI testing
 
 We use ui-testing in Miri, meaning we generate `.stderr` and `.stdout` files for the output
-produced by Miri. You can use `./miri bless` to automatically (re)generate these files when
+produced by Miri. You can use `./miri test --bless` to automatically (re)generate these files when
 you add new tests or change how Miri presents certain output.
 
 Note that when you also use `MIRIFLAGS` to change optimizations and similar, the ui output
@@ -116,7 +109,7 @@ to run the other checks while ignoring the ui output, use `MIRI_SKIP_UI_CHECKS=1
 
 For more info on how to configure ui tests see [the documentation on the ui test crate][ui_test]
 
-[ui_test]: ui_test/README.md
+[ui_test]: https://github.com/oli-obk/ui_test/blob/main/README.md
 
 ### Testing `cargo miri`
 
@@ -129,18 +122,15 @@ development version of Miri using
 ./miri install
 ```
 
-and then you can use it as if it was installed by `rustup`.  Make sure you use
-the same toolchain when calling `cargo miri` that you used when installing Miri!
-Usually this means you have to write `cargo +miri miri ...` to select the `miri`
-toolchain that was installed by `./miri toolchain`.
+and then you can use it as if it was installed by `rustup` as a component of the
+`miri` toolchain. Note that the `miri` and `cargo-miri` executables are placed
+in the `miri` toolchain's sysroot to prevent conflicts with other toolchains.
+The Miri binaries in the `cargo` bin directory (usually `~/.cargo/bin`) are managed by rustup.
 
 There's a test for the cargo wrapper in the `test-cargo-miri` directory; run
 `./run-test.py` in there to execute it. Like `./miri test`, this respects the
 `MIRI_TEST_TARGET` environment variable to execute the test for another target.
 
-Note that installing Miri like this will "take away" Miri management from `rustup`.
-If you want to later go back to a rustup-installed Miri, run `rustup update`.
-
 ### Using a modified standard library
 
 Miri re-builds the standard library into a custom sysroot, so it is fairly easy
@@ -168,16 +158,17 @@ to `.vscode/settings.json` in your local Miri clone:
 {
     "rust-analyzer.rustc.source": "discover",
     "rust-analyzer.linkedProjects": [
-        "./Cargo.toml",
-        "./cargo-miri/Cargo.toml"
+        "Cargo.toml",
+        "cargo-miri/Cargo.toml",
+        "miri-script/Cargo.toml",
     ],
-    "rust-analyzer.checkOnSave.overrideCommand": [
+    "rust-analyzer.check.overrideCommand": [
         "env",
         "MIRI_AUTO_OPS=no",
         "./miri",
         "cargo",
         "clippy", // make this `check` when working with a locally built rustc
-        "--message-format=json"
+        "--message-format=json",
     ],
     // Contrary to what the name suggests, this also affects proc macros.
     "rust-analyzer.cargo.buildScripts.overrideCommand": [
@@ -232,19 +223,17 @@ You can also directly run Miri on a Rust source file:
 
 ## Advanced topic: Syncing with the rustc repo
 
-We use the [`josh` proxy](https://github.com/josh-project/josh) to transmit
-changes between the rustc and Miri repositories. For now, the latest git version
-of josh needs to be built from source. This downloads and runs josh:
+We use the [`josh` proxy](https://github.com/josh-project/josh) to transmit changes between the
+rustc and Miri repositories. You can install it as follows:
 
 ```sh
-git clone https://github.com/josh-project/josh
-cd josh
-cargo run --release -p josh-proxy -- --local=local --remote=https://github.com --no-background
+cargo +stable install josh-proxy --git https://github.com/josh-project/josh --tag r22.12.06
 ```
 
+Josh will automatically be started and stopped by `./miri`.
+
 ### Importing changes from the rustc repo
 
-Josh needs to be running, as described above.
 We assume we start on an up-to-date master branch in the Miri repo.
 
 ```sh
@@ -263,16 +252,14 @@ needed.
 
 ### Exporting changes to the rustc repo
 
-Keep in mind that pushing is the most complicated job that josh has to do --
-pulling just filters the rustc history, but pushing needs to construct a new
-rustc history that would filter to the given Miri history! To avoid problems, it
-is a good idea to always pull immediately before you push. In particular, you
-should never do two josh pushes without an intermediate pull; that can lead to
-duplicated commits.
+Keep in mind that pushing is the most complicated job that josh has to do -- pulling just filters
+the rustc history, but pushing needs to construct a new rustc history that would filter to the given
+Miri history! To avoid problems, it is a good idea to always pull immediately before you push. If
+you are getting strange errors, chances are you are running into [this josh
+bug](https://github.com/josh-project/josh/issues/998). In that case, please get in touch on Zulip.
 
-Josh needs to be running, as described above. We will use the josh proxy to push
-to your fork of rustc. Run the following in the Miri repo, assuming we are on an
-up-to-date master branch:
+We will use the josh proxy to push to your fork of rustc. Run the following in the Miri repo,
+assuming we are on an up-to-date master branch:
 
 ```sh
 # Push the Miri changes to your rustc fork (substitute your github handle for YOUR_NAME).
@@ -282,3 +269,11 @@ up-to-date master branch:
 This will create a new branch called 'miri' in your fork, and the output should
 include a link to create a rustc PR that will integrate those changes into the
 main repository.
+
+If this fails due to authentication problems, it can help to make josh push via ssh instead of
+https. Add the following to your `.gitconfig`:
+
+```toml
+[url "git@github.com:"]
+    pushInsteadOf = https://github.com/
+```