Skip to content

Commit dabf72a

Browse files
authored
[CI] Fail benchmark CI only on regressions in CPU instruction count (#20277)
Benchmark script changes in this PR are NFC. Bit of a long overdue change: Wall time has proven to not be a good metric for stable performance numbers. This PR changes SYCL benchmarking CI regression filters to only fail on severe regressions in tests measuring CPU instruction count only. This PR does not affect the behavior of benchmarking scripts. It only affects the behavior of the CI w.r.t. regression failures for _SYCL_: it does not concern regressions detected in UR or L0.
1 parent f0e876a commit dabf72a

File tree

2 files changed

+6
-2
lines changed

2 files changed

+6
-2
lines changed

devops/actions/run-tests/benchmark/action.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -213,8 +213,8 @@ runs:
213213
--name "$SAVE_NAME" \
214214
--compare-file "./llvm-ci-perf-results/results/${SAVE_NAME}_${SAVE_TIMESTAMP}.json" \
215215
--results-dir "./llvm-ci-perf-results/results/" \
216-
--regression-filter '^[a-z_]+_sycl ' \
217-
--regression-filter-type 'SYCL' \
216+
--regression-filter '^[a-z_]+_sycl .* CPU count' \
217+
--regression-filter-type 'SYCL benchmark (measured using CPU cycle count)' \
218218
--verbose \
219219
--produce-github-summary \
220220
${{ inputs.dry_run == 'true' && '--dry-run' || '' }} \

devops/scripts/benchmarks/benches/compute.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -414,6 +414,10 @@ def run(
414414
ret = []
415415
for label, median, stddev, unit in parsed_results:
416416
extra_label = " CPU count" if parse_unit_type(unit) == "instr" else ""
417+
# Note: SYCL CI currently parses for on this "CPU count" value.
418+
# Please update /devops/scripts/benchmarks/compare.py if this value
419+
# is changed. See compare.py usage (w.r.t. --regression-filter) in
420+
# /devops/actions/run-tests/benchmarks/action.yml.
417421
ret.append(
418422
Result(
419423
label=self.name() + extra_label,

0 commit comments

Comments
 (0)