Run CI only for changed modules in pull requests #6610

smola · 2024-02-02T15:32:38Z

What Does This Do

Use AffectedModuleDetector gradle plugin to skip projects that are unaffected by git changes.
Only applied to PRs. Master/release branches run all tests. Using the label run-tests: all will also skip this logic and run all tests.
The current configuration considers that a project is affected if that project, or any dependency project (transitively) has changed.
Git changes are computed by comparing the merge checkout (that is, the code we actually test) against the latest commit in the base branch (by default, master).
Previous logic to skip instrumentation tests for profiler-only changes is kept intact, since there is an actual dependency across these modules which would always trigger all instrumentation tests. Ideally we would do some further changes to use a single skip mechanism for both cases.
Results: a PR pipeline currently takes 30-35min. A PR only touching a smoke test (one of the best cases), takes ~18min. Example PR with partial run: [TEST] Test CI diff changes #6620

Motivation

Reduce PR turnaround time, and reduce the chance of hitting a flaky test in a PR.

Additional Notes

To Do

Correctly select comparison base (master, release/).

Future work

Exclude irrelevant files (CI conf, README, etc)
Replace current profiler/debugging skip mechanisms
Avoid creating Circle CI jobs if none if its tasks will run (e.g. if no smoke test is run, setup should avoid creating smoke jobs at all, this might require a lockfile of project dependencies).
Add lockfile for project dependencies? Replace current profiler/debugging skip mechanism.

pr-commenter · 2024-02-02T16:21:52Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	smola/affected-modules
git_commit_date	1708014986	1708018305
git_commit_sha	`d57f54b`	`2fdce96`
release_version	1.31.0-SNAPSHOT~d57f54bf1a	1.31.0-SNAPSHOT~2fdce96cb7

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1708021250	1708021250
ci_job_id	435366154	435366154
ci_pipeline_id	28372318	28372318
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 43 metrics, 11 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.31.0-SNAPSHOT~2fdce96cb7, baseline=1.31.0-SNAPSHOT~d57f54bf1a

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.071 s) : 0, 1070788
Total [baseline] (9.341 s) : 0, 9340747
Agent [candidate] (1.064 s) : 0, 1064055
Total [candidate] (9.378 s) : 0, 9378057
section appsec
Agent [baseline] (1.159 s) : 0, 1158782
Total [baseline] (9.442 s) : 0, 9442022
Agent [candidate] (1.169 s) : 0, 1168595
Total [candidate] (9.439 s) : 0, 9438831
section iast
Agent [baseline] (1.198 s) : 0, 1198000
Total [baseline] (9.715 s) : 0, 9714546
Agent [candidate] (1.187 s) : 0, 1187360
Total [candidate] (9.702 s) : 0, 9701521
section profiling
Agent [baseline] (1.278 s) : 0, 1278086
Total [baseline] (9.616 s) : 0, 9615938
Agent [candidate] (1.28 s) : 0, 1279991
Total [candidate] (9.545 s) : 0, 9544938

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.071 s	-
Agent	appsec	1.159 s	87.994 ms (8.2%)
Agent	iast	1.198 s	127.212 ms (11.9%)
Agent	profiling	1.278 s	207.298 ms (19.4%)
Total	tracing	9.341 s	-
Total	appsec	9.442 s	101.276 ms (1.1%)
Total	iast	9.715 s	373.799 ms (4.0%)
Total	profiling	9.616 s	275.191 ms (2.9%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.064 s	-
Agent	appsec	1.169 s	104.54 ms (9.8%)
Agent	iast	1.187 s	123.305 ms (11.6%)
Agent	profiling	1.28 s	215.936 ms (20.3%)
Total	tracing	9.378 s	-
Total	appsec	9.439 s	60.774 ms (0.6%)
Total	iast	9.702 s	323.464 ms (3.4%)
Total	profiling	9.545 s	166.881 ms (1.8%)

gantt
    title petclinic - break down per module: candidate=1.31.0-SNAPSHOT~2fdce96cb7, baseline=1.31.0-SNAPSHOT~d57f54bf1a

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (673.964 ms) : 0, 673964
BytebuddyAgent [candidate] (669.564 ms) : 0, 669564
GlobalTracer [baseline] (301.89 ms) : 0, 301890
GlobalTracer [candidate] (299.943 ms) : 0, 299943
AppSec [baseline] (51.877 ms) : 0, 51877
AppSec [candidate] (51.782 ms) : 0, 51782
Remote Config [baseline] (704.774 µs) : 0, 705
Remote Config [candidate] (701.2 µs) : 0, 701
Telemetry [baseline] (7.756 ms) : 0, 7756
Telemetry [candidate] (7.672 ms) : 0, 7672
section appsec
BytebuddyAgent [baseline] (667.868 ms) : 0, 667868
BytebuddyAgent [candidate] (673.292 ms) : 0, 673292
GlobalTracer [baseline] (298.307 ms) : 0, 298307
GlobalTracer [candidate] (301.561 ms) : 0, 301561
AppSec [baseline] (150.937 ms) : 0, 150937
AppSec [candidate] (151.629 ms) : 0, 151629
Remote Config [baseline] (634.718 µs) : 0, 635
Remote Config [candidate] (643.808 µs) : 0, 644
Telemetry [baseline] (6.808 ms) : 0, 6808
Telemetry [candidate] (6.905 ms) : 0, 6905
section iast
BytebuddyAgent [baseline] (789.743 ms) : 0, 789743
BytebuddyAgent [candidate] (780.024 ms) : 0, 780024
GlobalTracer [baseline] (291.266 ms) : 0, 291266
GlobalTracer [candidate] (290.393 ms) : 0, 290393
AppSec [baseline] (51.62 ms) : 0, 51620
AppSec [candidate] (54.609 ms) : 0, 54609
Remote Config [baseline] (658.236 µs) : 0, 658
Remote Config [candidate] (625.188 µs) : 0, 625
Telemetry [baseline] (6.64 ms) : 0, 6640
Telemetry [candidate] (6.585 ms) : 0, 6585
IAST [baseline] (23.275 ms) : 0, 23275
IAST [candidate] (20.758 ms) : 0, 20758
section profiling
BytebuddyAgent [baseline] (665.6 ms) : 0, 665600
BytebuddyAgent [candidate] (667.183 ms) : 0, 667183
GlobalTracer [baseline] (381.082 ms) : 0, 381082
GlobalTracer [candidate] (382.965 ms) : 0, 382965
AppSec [baseline] (52.272 ms) : 0, 52272
AppSec [candidate] (52.068 ms) : 0, 52068
Remote Config [baseline] (662.281 µs) : 0, 662
Remote Config [candidate] (656.339 µs) : 0, 656
Telemetry [baseline] (11.749 ms) : 0, 11749
Telemetry [candidate] (10.255 ms) : 0, 10255
ProfilingAgent [baseline] (111.992 ms) : 0, 111992
ProfilingAgent [candidate] (112.102 ms) : 0, 112102
Profiling [baseline] (112.017 ms) : 0, 112017
Profiling [candidate] (112.126 ms) : 0, 112126

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2024-02-15T17:56:56	2024-02-15T18:15:49
git_branch	master	smola/affected-modules
git_commit_date	1708014986	1708018305
git_commit_sha	`d57f54b`	`2fdce96`
release_version	1.31.0-SNAPSHOT~d57f54bf1a	1.31.0-SNAPSHOT~2fdce96cb7
start_time	2024-02-15T17:56:43	2024-02-15T18:15:36

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1708021250	1708021250
ci_job_id	435366154	435366154
ci_pipeline_id	28372318	28372318
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 15 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.31.0-SNAPSHOT~2fdce96cb7, baseline=1.31.0-SNAPSHOT~d57f54bf1a
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.351 ms) : 1333, 1370
.   : milestone, 1351,
appsec (1.782 ms) : 1757, 1808
.   : milestone, 1782,
iast (1.528 ms) : 1503, 1552
.   : milestone, 1528,
profiling (1.518 ms) : 1492, 1543
.   : milestone, 1518,
tracing (1.514 ms) : 1489, 1539
.   : milestone, 1514,
section candidate
no_agent (1.345 ms) : 1326, 1364
.   : milestone, 1345,
appsec (1.764 ms) : 1739, 1789
.   : milestone, 1764,
iast (1.531 ms) : 1507, 1555
.   : milestone, 1531,
profiling (1.537 ms) : 1511, 1562
.   : milestone, 1537,
tracing (1.492 ms) : 1467, 1517
.   : milestone, 1492,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.351 ms [1.333 ms, 1.37 ms]	-
appsec	1.782 ms [1.757 ms, 1.808 ms]	430.798 µs (31.9%)
iast	1.528 ms [1.503 ms, 1.552 ms]	176.238 µs (13.0%)
profiling	1.518 ms [1.492 ms, 1.543 ms]	166.043 µs (12.3%)
tracing	1.514 ms [1.489 ms, 1.539 ms]	162.395 µs (12.0%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.345 ms [1.326 ms, 1.364 ms]	-
appsec	1.764 ms [1.739 ms, 1.789 ms]	418.616 µs (31.1%)
iast	1.531 ms [1.507 ms, 1.555 ms]	185.619 µs (13.8%)
profiling	1.537 ms [1.511 ms, 1.562 ms]	191.466 µs (14.2%)
tracing	1.492 ms [1.467 ms, 1.517 ms]	146.747 µs (10.9%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.31.0-SNAPSHOT~2fdce96cb7, baseline=1.31.0-SNAPSHOT~d57f54bf1a
    dateFormat X
    axisFormat %s
section baseline
no_agent (368.776 µs) : 348, 389
.   : milestone, 369,
iast (471.057 µs) : 450, 492
.   : milestone, 471,
iast_FULL (538.966 µs) : 518, 560
.   : milestone, 539,
iast_GLOBAL (493.893 µs) : 473, 515
.   : milestone, 494,
iast_HARDCODED_SECRET_DISABLED (478.312 µs) : 457, 500
.   : milestone, 478,
iast_INACTIVE (445.536 µs) : 425, 466
.   : milestone, 446,
iast_TELEMETRY_OFF (468.844 µs) : 447, 490
.   : milestone, 469,
tracing (442.884 µs) : 421, 464
.   : milestone, 443,
section candidate
no_agent (365.778 µs) : 346, 386
.   : milestone, 366,
iast (475.854 µs) : 454, 497
.   : milestone, 476,
iast_FULL (534.408 µs) : 514, 555
.   : milestone, 534,
iast_GLOBAL (490.927 µs) : 471, 511
.   : milestone, 491,
iast_HARDCODED_SECRET_DISABLED (474.923 µs) : 454, 496
.   : milestone, 475,
iast_INACTIVE (439.512 µs) : 419, 460
.   : milestone, 440,
iast_TELEMETRY_OFF (469.494 µs) : 449, 490
.   : milestone, 469,
tracing (442.758 µs) : 422, 463
.   : milestone, 443,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	368.776 µs [348.421 µs, 389.131 µs]	-
iast	471.057 µs [450.455 µs, 491.66 µs]	102.281 µs (27.7%)
iast_FULL	538.966 µs [517.73 µs, 560.202 µs]	170.19 µs (46.1%)
iast_GLOBAL	493.893 µs [472.544 µs, 515.241 µs]	125.116 µs (33.9%)
iast_HARDCODED_SECRET_DISABLED	478.312 µs [456.671 µs, 499.953 µs]	109.536 µs (29.7%)
iast_INACTIVE	445.536 µs [424.875 µs, 466.196 µs]	76.76 µs (20.8%)
iast_TELEMETRY_OFF	468.844 µs [447.427 µs, 490.262 µs]	100.068 µs (27.1%)
tracing	442.884 µs [421.362 µs, 464.406 µs]	74.108 µs (20.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	365.778 µs [345.821 µs, 385.736 µs]	-
iast	475.854 µs [454.464 µs, 497.245 µs]	110.076 µs (30.1%)
iast_FULL	534.408 µs [514.083 µs, 554.733 µs]	168.63 µs (46.1%)
iast_GLOBAL	490.927 µs [470.536 µs, 511.317 µs]	125.149 µs (34.2%)
iast_HARDCODED_SECRET_DISABLED	474.923 µs [454.171 µs, 495.676 µs]	109.145 µs (29.8%)
iast_INACTIVE	439.512 µs [418.926 µs, 460.097 µs]	73.733 µs (20.2%)
iast_TELEMETRY_OFF	469.494 µs [448.518 µs, 490.47 µs]	103.716 µs (28.4%)
tracing	442.758 µs [422.287 µs, 463.229 µs]	76.98 µs (21.0%)

bantonsson · 2024-02-06T16:21:53Z

.circleci/render_config.py

            )
            resp.raise_for_status()
        except Exception as e:
-            print(f"Request filed: {e}")


bantonsson · 2024-02-06T16:23:37Z

build.gradle

 buildscript {
  dependencies {
    classpath "pl.allegro.tech.build:axion-release-plugin:1.14.4"
+    classpath "com.dropbox.affectedmoduledetector:affectedmoduledetector:0.2.2"


I had to look, and it doesn't use jgit 🎉

This reverts commit f66853f.

* Revert "Fix setup job in CI (#6688)" This reverts commit 85a5f5a. * Revert "Run CI only for changed modules in pull requests (#6610)" This reverts commit f66853f.

smola added tag: do not merge Do not merge changes comp: testing Testing tag: no release notes Changes to exclude from release notes labels Feb 2, 2024

smola force-pushed the smola/affected-modules branch 5 times, most recently from 2841eec to 30166b9 Compare February 2, 2024 16:08

smola force-pushed the smola/affected-modules branch 7 times, most recently from 7b52719 to e88e5d0 Compare February 6, 2024 15:00

Run CI only for changed modules in pull requests

436220e

smola force-pushed the smola/affected-modules branch from e88e5d0 to 436220e Compare February 6, 2024 15:01

smola marked this pull request as ready for review February 6, 2024 15:47

smola requested a review from a team as a code owner February 6, 2024 15:47

smola requested review from dougqh and ygree February 6, 2024 15:47

bantonsson approved these changes Feb 6, 2024

View reviewed changes

smola removed the tag: do not merge Do not merge changes label Feb 7, 2024

smola added 4 commits February 8, 2024 10:28

Merge branch 'master' into smola/affected-modules

2e28b1f

Merge branch 'master' into smola/affected-modules

1970a21

Merge branch 'master' into smola/affected-modules

dd21fa1

Merge branch 'master' into smola/affected-modules

2fdce96

smola merged commit f66853f into master Feb 16, 2024

smola deleted the smola/affected-modules branch February 16, 2024 07:39

github-actions bot added this to the 1.31.0 milestone Feb 16, 2024

smola mentioned this pull request Feb 16, 2024

Fix setup job in CI #6688

Merged

smola added a commit that referenced this pull request Feb 19, 2024

Revert "Run CI only for changed modules in pull requests (#6610)"

3ccf85e

This reverts commit f66853f.

smola mentioned this pull request Feb 19, 2024

Revert "Run CI only for changed modules in pull requests #6610" #6699

Merged

smola mentioned this pull request Aug 19, 2024

Run CI only for tasks affected by git changes #7464

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Run CI only for changed modules in pull requests #6610

Run CI only for changed modules in pull requests #6610

Uh oh!

smola commented Feb 2, 2024 •

edited

Loading

Uh oh!

pr-commenter bot commented Feb 2, 2024 •

edited

Loading

Uh oh!

bantonsson Feb 6, 2024

Uh oh!

bantonsson Feb 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Run CI only for changed modules in pull requests #6610

Run CI only for changed modules in pull requests #6610

Uh oh!

Conversation

smola commented Feb 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

To Do

Future work

Uh oh!

pr-commenter bot commented Feb 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Uh oh!

bantonsson Feb 6, 2024

Choose a reason for hiding this comment

Uh oh!

bantonsson Feb 6, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

smola commented Feb 2, 2024 •

edited

Loading

pr-commenter bot commented Feb 2, 2024 •

edited

Loading