ASV custom build command and file-based benchmark triggers (#5776)

* Trigger benchmarks if lock files have changed.

* Check for 'true' string not boolean

* Temporarily disable other CI.

* Limit branch actions to pull requests

* Unified continue check.

* Benchmarks temporary debug mode.

* Demonstrate lock file trigger for benchmarks.

* Demonstrate overnight mode.

* Revert "Demonstrate overnight mode."

This reverts commit fbb4ab7.

* Revert "Demonstrate lock file trigger for benchmarks."

This reverts commit 5522251.

* Minor visual improvements.

* Explanatory notes.

* Revert "Benchmarks temporary debug mode."

This reverts commit 8f097b1.

* Revert "Temporarily disable other CI."

This reverts commit 621a774.

* Fix docs indentation.

* Custom ASV build command using setup.py.

* Remove ASV pin.

* Don't trigger benchmarks when pyproject.toml changes.

* More accurate name for files-changed step.

* Don't have a continue output.

* Remove confusing variable name clash.

* Update What's New entry.

Author: trexfeathers

.github/labeler.yml

@@ -1,6 +1,9 @@
-# Make sure any dependency changes are benchmarked (only changes to the locks
-#  make a material difference - changes to the Conda YAML files are not
-#  benchmarked).
+# benchmark_this automatically triggers the benchmark workflow when added by
+#  a user. No triggering happens when GitHub Actions adds the label (this
+#  avoids security vulnerabilities), so alternative triggers for the below
+#  files are therefore included in workflows/benchmarks_run.yml. Automatic
+#  labelling is still included here to make it easier to search pull requests,
+#  and to reinforce the culture of using this label.
 benchmark_this:
 - changed-files:
-  - any-glob-to-any-file: 'requirements/locks/*.lock'
+  - any-glob-to-any-file: ['requirements/locks/*.lock', "setup.py"]

.github/pull_request_template.md

@@ -7,3 +7,8 @@
 
 ---
 [Consult Iris pull request check list]( https://scitools-iris.readthedocs.io/en/latest/developers_guide/contributing_pull_request_checklist.html)
+
+---
+Add any of the below labels to trigger actions on this PR:
+
+- https://github.com/SciTools/iris/labels/benchmark_this

.github/workflows/benchmarks_run.yml

@@ -16,15 +16,46 @@ on:
         required: false
         type: string
   pull_request:
-    types: [labeled]
+    # Add the `labeled` type to the default list.
+    types: [labeled, opened, synchronize, reopened]
 
 jobs:
+  pre-checks:
+    runs-on: ubuntu-latest
+    if: github.repository == 'SciTools/iris'
+    outputs:
+      overnight: ${{ steps.overnight.outputs.check }}
+      branch: ${{ steps.branch.outputs.check }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 2
+      - id: files-changed
+        uses: marceloprado/has-changed-path@df1b7a3161b8fb9fd8c90403c66a9e66dfde50cb
+        with:
+          # SEE ALSO .github/labeler.yml .
+          paths: requirements/locks/*.lock setup.py
+      - id: overnight
+        if: github.event_name != 'pull_request'
+        run: echo "check=true" >> "$GITHUB_OUTPUT"
+      - id: branch
+        if: >
+          github.event_name == 'pull_request'
+          &&
+          (
+            steps.files-changed.outputs.changed == 'true'
+            ||
+            github.event.label.name == 'benchmark_this'
+          )
+        run: echo "check=true" >> "$GITHUB_OUTPUT"
+
+
   benchmark:
-    if: >
-      github.repository == 'SciTools/iris' && 
-      (github.event_name != 'pull_request' || 
-      github.event.label.name == 'benchmark_this')
     runs-on: ubuntu-latest
+    needs: pre-checks
+    if: >
+      needs.pre-checks.outputs.overnight == 'true' ||
+      needs.pre-checks.outputs.branch == 'true'
 
     env:
       IRIS_TEST_DATA_LOC_PATH: benchmarks
@@ -42,7 +73,7 @@ jobs:
 
       - name: Install ASV & Nox
         run: |
-          pip install "asv!=0.6.2" nox
+          pip install asv nox
 
       - name: Cache environment directories
         id: cache-env-dir
@@ -76,7 +107,7 @@ jobs:
           echo "OVERRIDE_TEST_DATA_REPOSITORY=${GITHUB_WORKSPACE}/${IRIS_TEST_DATA_PATH}/test_data" >> $GITHUB_ENV
 
       - name: Benchmark this pull request
-        if: ${{ github.event.label.name == 'benchmark_this' }}
+        if: needs.pre-checks.outputs.branch == 'true'
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           PR_NUMBER: ${{ github.event.number }}
@@ -85,7 +116,7 @@ jobs:
 
       - name: Run overnight benchmarks
         id: overnight
-        if: ${{ github.event_name != 'pull_request' }}
+        if: needs.pre-checks.outputs.overnight == 'true'
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |

benchmarks/asv.conf.json

@@ -6,6 +6,10 @@
     "environment_type": "conda-delegated",
     "show_commit_url": "https://github.com/scitools/iris/commit/",
     "branches": ["upstream/main"],
+    "build_command": [
+        "python setup.py build",
+        "python -mpip wheel --no-deps -w {build_cache_dir} {build_dir}"
+    ],
 
     "benchmark_dir": "./benchmarks",
     "env_dir": ".asv/env",

benchmarks/bm_runner.py

@@ -171,7 +171,7 @@ def _gh_create_reports(commit_sha: str, results_full: str, results_shifts: str)
     performance_report = dedent(
         (
             """
-            ### Performance Benchmark Report: {commit_sha}
+            # :stopwatch: Performance Benchmark Report: {commit_sha}
 
             <details>
             <summary>Performance shifts</summary>

docs/src/whatsnew/latest.rst

@@ -72,11 +72,10 @@ This document explains the changes made to Iris for this release
 💼 Internal
 ===========
 
-#. `@trexfeathers`_ used the `Pull Request Labeler Github action`_ to add the
-   ``benchmark_this`` label (:ref:`more info <on_demand_pr_benchmark>`) to
-   pull requests that modify ``requirements/locks/*.lock`` files - ensuring
-   that we know whether dependency changes will affect performance.
-   (:pull:`5763`)
+#. `@trexfeathers`_ setup automatic benchmarking on pull requests that modify
+   files likely to affect performance or performance testing. Such pull
+   requests are also labelled using the `Pull Request Labeler Github action`_
+   to increase visibility. (:pull:`5763`, :pull:`5776`)
 
 
 .. comment