Skip to content

Run CI only for tasks affected by git changes #7464

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
Aug 21, 2024
Merged

Run CI only for tasks affected by git changes #7464

merged 7 commits into from
Aug 21, 2024

Conversation

@smola
Copy link
Member

@smola smola commented Aug 18, 2024 •
edited
Loading

What Does This Do

  • Add a new property -PgitBaseRef that can be set to a git reference (or branch, tag, commit hash, etc) to compare the current HEAD against. This will compute which files changed. Changed our Circle CI configuration to pass -PgitBaseRef=<base branch> for PRs.
  • Aggregate tasks like :baseTests, :instTests, :baseCheck, etc, now do not trigger tasks that are unaffected by the computed git changes.
  • The logic to find affected tasks is as follows:
    • Get changes from git (e.g. git diff --names-only origin/master..HEAD).
    • Ignore certain files, such as README.md. Check the definition of ignoredFiles for the current patterns.
    • Detect global effect files, such as gradle/**. If any of these changes, the rest of the change tracking logic will be skipped and all tasks will be triggered. Check the definition of globalEffectFiles.
    • For each file, infer the project and task it should trigger directly. A file will be assigned to the project with the longest path which is a prefix of the file. If this cannot be found (e.g. root project changes), change tracking is disabled, and all tasks will be triggered.
    • Once a project is identified, a file will affect a set of tasks. If it's in src/test, it'll affect testClasses. If it's in src/testFixtures, it'll affect testFixturesClasses. If it's in src/jmh, it'll affect jmhCompileGeneratedClasses. Otherwise, it'll affect all tasks in the project.
    • Given an aggregate task (e.g. :baseTests), get all relevant tasks, after applying partitioning if necessary. And for each task, that task and all its transitive task dependencies will be checked against the previous collected list of affected tasks. If an affected task is found, then the task will be triggered.

Note that logging here is quite verbose, using warning level. The intent is making any decision around change tracking as transparent and debuggeable as possible.

Some minor notes around CI definition:

  • Previous logic to skip instrumentation tests for profiler-only changes is kept intact, since there is an actual dependency across these modules which would always trigger all instrumentation tests. Ideally we would do some further changes to use a single skip mechanism for both cases.
  • dd-java-agent/agent-iast does not skip instrumentation tests anymore, since there's a few instrumentation modules that depend on test fixtures from dd-java-agent/agent-iast.
  • .circleci/collect_results.sh was changed to not fail when no tests were run in a job and no JUnit XML reports are available. This is necessary when there are no affected task for a given job partition.

How to test it?

To test change detection, you can use -PgitNewRef to simulate change detection as if another reference was checked out. For example, to see which tasks would be run for :baseTest on a given branch when compared to master, the following command can be used:

./gradlew :baseTest -PgitBaseRef=origin/master -PgitNewRef=my-branch

Motivation

Reduce PR turnaround time, and reduce the chance of hitting a flaky test in a PR.

Additional Notes

Contributor Checklist

@github-actions
Copy link
Contributor

Hi! 👋 Looks like you updated a Git Submodule.
If this was not intentional please make sure to:

@smola smola force-pushed the smola/git-changes-test branch from b203f79 to def6068 Compare August 18, 2024 16:28
@pr-commenter
Copy link

pr-commenter bot commented Aug 18, 2024 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/git-changes-test
git_commit_date 1724168948 1724170688
git_commit_sha 63ccd4c a89f547
release_version 1.39.0-SNAPSHOT~63ccd4c8fc 1.39.0-SNAPSHOT~a89f5471dc
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1724173159 1724173159
ci_job_id 610844147 610844147
ci_pipeline_id 42295767 42295767
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 45 metrics, 18 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.39.0-SNAPSHOT~a89f5471dc, baseline=1.39.0-SNAPSHOT~63ccd4c8fc

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.047 s) : 0, 1046563
Total [baseline] (10.367 s) : 0, 10367168
Agent [candidate] (1.046 s) : 0, 1046351
Total [candidate] (10.459 s) : 0, 10458875
section appsec
Agent [baseline] (1.168 s) : 0, 1167829
Total [baseline] (10.441 s) : 0, 10441143
Agent [candidate] (1.175 s) : 0, 1174651
Total [candidate] (10.468 s) : 0, 10467860
section iast
Agent [baseline] (1.185 s) : 0, 1185142
Total [baseline] (10.883 s) : 0, 10882748
Agent [candidate] (1.177 s) : 0, 1177036
Total [candidate] (10.844 s) : 0, 10843510
section profiling
Agent [baseline] (1.243 s) : 0, 1243484
Total [baseline] (10.564 s) : 0, 10564297
Agent [candidate] (1.244 s) : 0, 1243758
Total [candidate] (10.516 s) : 0, 10516210
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.047 s -
Agent appsec 1.168 s 121.267 ms (11.6%)
Agent iast 1.185 s 138.579 ms (13.2%)
Agent profiling 1.243 s 196.921 ms (18.8%)
Total tracing 10.367 s -
Total appsec 10.441 s 73.975 ms (0.7%)
Total iast 10.883 s 515.58 ms (5.0%)
Total profiling 10.564 s 197.129 ms (1.9%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.046 s -
Agent appsec 1.175 s 128.299 ms (12.3%)
Agent iast 1.177 s 130.685 ms (12.5%)
Agent profiling 1.244 s 197.407 ms (18.9%)
Total tracing 10.459 s -
Total appsec 10.468 s 8.985 ms (0.1%)
Total iast 10.844 s 384.635 ms (3.7%)
Total profiling 10.516 s 57.335 ms (0.5%) 
gantt
    title petclinic - break down per module: candidate=1.39.0-SNAPSHOT~a89f5471dc, baseline=1.39.0-SNAPSHOT~63ccd4c8fc

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (668.034 ms) : 0, 668034
BytebuddyAgent [candidate] (667.187 ms) : 0, 667187
GlobalTracer [baseline] (305.805 ms) : 0, 305805
GlobalTracer [candidate] (306.417 ms) : 0, 306417
AppSec [baseline] (51.196 ms) : 0, 51196
AppSec [candidate] (51.159 ms) : 0, 51159
Remote Config [baseline] (682.113 µs) : 0, 682
Remote Config [candidate] (695.705 µs) : 0, 696
Telemetry [baseline] (7.424 ms) : 0, 7424
Telemetry [candidate] (7.446 ms) : 0, 7446
section appsec
BytebuddyAgent [baseline] (677.637 ms) : 0, 677637
BytebuddyAgent [candidate] (681.985 ms) : 0, 681985
GlobalTracer [baseline] (299.406 ms) : 0, 299406
GlobalTracer [candidate] (301.271 ms) : 0, 301271
AppSec [baseline] (157.299 ms) : 0, 157299
AppSec [candidate] (157.906 ms) : 0, 157906
Remote Config [baseline] (600.49 µs) : 0, 600
Remote Config [candidate] (609.346 µs) : 0, 609
Telemetry [baseline] (8.733 ms) : 0, 8733
Telemetry [candidate] (8.806 ms) : 0, 8806
IAST [baseline] (21.425 ms) : 0, 21425
IAST [candidate] (21.04 ms) : 0, 21040
section iast
BytebuddyAgent [baseline] (788.727 ms) : 0, 788727
BytebuddyAgent [candidate] (783.14 ms) : 0, 783140
GlobalTracer [baseline] (298.398 ms) : 0, 298398
GlobalTracer [candidate] (296.766 ms) : 0, 296766
AppSec [baseline] (53.373 ms) : 0, 53373
AppSec [candidate] (53.194 ms) : 0, 53194
Remote Config [baseline] (583.216 µs) : 0, 583
Remote Config [candidate] (599.91 µs) : 0, 600
Telemetry [baseline] (8.79 ms) : 0, 8790
Telemetry [candidate] (7.955 ms) : 0, 7955
IAST [baseline] (21.702 ms) : 0, 21702
IAST [candidate] (21.896 ms) : 0, 21896
section profiling
ProfilingAgent [baseline] (94.485 ms) : 0, 94485
ProfilingAgent [candidate] (94.75 ms) : 0, 94750
BytebuddyAgent [baseline] (662.59 ms) : 0, 662590
BytebuddyAgent [candidate] (662.491 ms) : 0, 662491
GlobalTracer [baseline] (389.149 ms) : 0, 389149
GlobalTracer [candidate] (389.016 ms) : 0, 389016
AppSec [baseline] (52.047 ms) : 0, 52047
AppSec [candidate] (52.308 ms) : 0, 52308
Remote Config [baseline] (693.107 µs) : 0, 693
Remote Config [candidate] (675.768 µs) : 0, 676
Telemetry [baseline] (7.413 ms) : 0, 7413
Telemetry [candidate] (7.361 ms) : 0, 7361
Profiling [baseline] (94.509 ms) : 0, 94509
Profiling [candidate] (94.776 ms) : 0, 94776
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.39.0-SNAPSHOT~a89f5471dc, baseline=1.39.0-SNAPSHOT~63ccd4c8fc

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.057 s) : 0, 1057173
Total [baseline] (8.509 s) : 0, 8508764
Agent [candidate] (1.048 s) : 0, 1047674
Total [candidate] (8.486 s) : 0, 8486230
section iast
Agent [baseline] (1.174 s) : 0, 1173642
Total [baseline] (8.983 s) : 0, 8982602
Agent [candidate] (1.177 s) : 0, 1176610
Total [candidate] (8.975 s) : 0, 8975286
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.182 s) : 0, 1182174
Total [baseline] (8.966 s) : 0, 8965641
Agent [candidate] (1.191 s) : 0, 1191198
Total [candidate] (9.01 s) : 0, 9009556
section iast_TELEMETRY_OFF
Agent [baseline] (1.18 s) : 0, 1180364
Total [baseline] (9.006 s) : 0, 9006073
Agent [candidate] (1.171 s) : 0, 1171351
Total [candidate] (9.003 s) : 0, 9002621
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.057 s -
Agent iast 1.174 s 116.469 ms (11.0%)
Agent iast_HARDCODED_SECRET_DISABLED 1.182 s 125.001 ms (11.8%)
Agent iast_TELEMETRY_OFF 1.18 s 123.191 ms (11.7%)
Total tracing 8.509 s -
Total iast 8.983 s 473.839 ms (5.6%)
Total iast_HARDCODED_SECRET_DISABLED 8.966 s 456.877 ms (5.4%)
Total iast_TELEMETRY_OFF 9.006 s 497.309 ms (5.8%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.048 s -
Agent iast 1.177 s 128.936 ms (12.3%)
Agent iast_HARDCODED_SECRET_DISABLED 1.191 s 143.524 ms (13.7%)
Agent iast_TELEMETRY_OFF 1.171 s 123.677 ms (11.8%)
Total tracing 8.486 s -
Total iast 8.975 s 489.056 ms (5.8%)
Total iast_HARDCODED_SECRET_DISABLED 9.01 s 523.326 ms (6.2%)
Total iast_TELEMETRY_OFF 9.003 s 516.391 ms (6.1%) 
gantt
    title insecure-bank - break down per module: candidate=1.39.0-SNAPSHOT~a89f5471dc, baseline=1.39.0-SNAPSHOT~63ccd4c8fc

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (674.852 ms) : 0, 674852
BytebuddyAgent [candidate] (668.162 ms) : 0, 668162
GlobalTracer [baseline] (309.228 ms) : 0, 309228
GlobalTracer [candidate] (306.869 ms) : 0, 306869
AppSec [baseline] (51.291 ms) : 0, 51291
AppSec [candidate] (51.053 ms) : 0, 51053
Remote Config [baseline] (690.135 µs) : 0, 690
Remote Config [candidate] (677.433 µs) : 0, 677
Telemetry [baseline] (7.529 ms) : 0, 7529
Telemetry [candidate] (7.44 ms) : 0, 7440
section iast
BytebuddyAgent [baseline] (780.407 ms) : 0, 780407
BytebuddyAgent [candidate] (782.918 ms) : 0, 782918
GlobalTracer [baseline] (295.209 ms) : 0, 295209
GlobalTracer [candidate] (296.357 ms) : 0, 296357
AppSec [baseline] (50.123 ms) : 0, 50123
AppSec [candidate] (53.877 ms) : 0, 53877
IAST [baseline] (25.131 ms) : 0, 25131
IAST [candidate] (22.383 ms) : 0, 22383
Remote Config [baseline] (603.05 µs) : 0, 603
Remote Config [candidate] (559.47 µs) : 0, 559
Telemetry [baseline] (8.727 ms) : 0, 8727
Telemetry [candidate] (7.057 ms) : 0, 7057
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (786.319 ms) : 0, 786319
BytebuddyAgent [candidate] (791.421 ms) : 0, 791421
GlobalTracer [baseline] (297.837 ms) : 0, 297837
GlobalTracer [candidate] (299.928 ms) : 0, 299928
AppSec [baseline] (50.412 ms) : 0, 50412
AppSec [candidate] (51.115 ms) : 0, 51115
IAST [baseline] (24.808 ms) : 0, 24808
IAST [candidate] (23.983 ms) : 0, 23983
Remote Config [baseline] (566.7 µs) : 0, 567
Remote Config [candidate] (625.784 µs) : 0, 626
Telemetry [baseline] (8.669 ms) : 0, 8669
Telemetry [candidate] (10.511 ms) : 0, 10511
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (784.995 ms) : 0, 784995
BytebuddyAgent [candidate] (778.977 ms) : 0, 778977
GlobalTracer [baseline] (298.651 ms) : 0, 298651
GlobalTracer [candidate] (296.597 ms) : 0, 296597
AppSec [baseline] (50.528 ms) : 0, 50528
AppSec [candidate] (51.33 ms) : 0, 51330
IAST [baseline] (24.159 ms) : 0, 24159
IAST [candidate] (23.194 ms) : 0, 23194
Remote Config [baseline] (619.198 µs) : 0, 619
Remote Config [candidate] (626.883 µs) : 0, 627
Telemetry [baseline] (7.852 ms) : 0, 7852
Telemetry [candidate] (7.102 ms) : 0, 7102
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-08-20T16:30:01 2024-08-20T16:36:51
git_branch master smola/git-changes-test
git_commit_date 1724168948 1724170688
git_commit_sha 63ccd4c a89f547
release_version 1.39.0-SNAPSHOT~63ccd4c8fc 1.39.0-SNAPSHOT~a89f5471dc
start_time 2024-08-20T16:29:48 2024-08-20T16:36:38
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1724172157 1724172157
ci_job_id 610844148 610844148
ci_pipeline_id 42295767 42295767
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.39.0-SNAPSHOT~a89f5471dc, baseline=1.39.0-SNAPSHOT~63ccd4c8fc
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.336 ms) : 1317, 1356
.   : milestone, 1336,
appsec (1.728 ms) : 1704, 1753
.   : milestone, 1728,
appsec_no_iast (1.723 ms) : 1699, 1747
.   : milestone, 1723,
iast (1.465 ms) : 1441, 1488
.   : milestone, 1465,
profiling (1.513 ms) : 1487, 1539
.   : milestone, 1513,
tracing (1.467 ms) : 1443, 1492
.   : milestone, 1467,
section candidate
no_agent (1.339 ms) : 1319, 1358
.   : milestone, 1339,
appsec (1.735 ms) : 1712, 1759
.   : milestone, 1735,
appsec_no_iast (1.738 ms) : 1713, 1762
.   : milestone, 1738,
iast (1.478 ms) : 1456, 1500
.   : milestone, 1478,
profiling (1.568 ms) : 1543, 1594
.   : milestone, 1568,
tracing (1.446 ms) : 1421, 1471
.   : milestone, 1446,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.336 ms [1.317 ms, 1.356 ms] -
appsec 1.728 ms [1.704 ms, 1.753 ms] 392.12 µs (29.3%)
appsec_no_iast 1.723 ms [1.699 ms, 1.747 ms] 387.166 µs (29.0%)
iast 1.465 ms [1.441 ms, 1.488 ms] 128.276 µs (9.6%)
profiling 1.513 ms [1.487 ms, 1.539 ms] 176.726 µs (13.2%)
tracing 1.467 ms [1.443 ms, 1.492 ms] 131.078 µs (9.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.339 ms [1.319 ms, 1.358 ms] -
appsec 1.735 ms [1.712 ms, 1.759 ms] 396.907 µs (29.7%)
appsec_no_iast 1.738 ms [1.713 ms, 1.762 ms] 398.959 µs (29.8%)
iast 1.478 ms [1.456 ms, 1.5 ms] 139.129 µs (10.4%)
profiling 1.568 ms [1.543 ms, 1.594 ms] 229.832 µs (17.2%)
tracing 1.446 ms [1.421 ms, 1.471 ms] 107.381 µs (8.0%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.39.0-SNAPSHOT~a89f5471dc, baseline=1.39.0-SNAPSHOT~63ccd4c8fc
    dateFormat X
    axisFormat %s
section baseline
no_agent (372.444 µs) : 352, 392
.   : milestone, 372,
iast (480.659 µs) : 459, 502
.   : milestone, 481,
iast_FULL (547.126 µs) : 525, 569
.   : milestone, 547,
iast_GLOBAL (502.352 µs) : 480, 525
.   : milestone, 502,
iast_HARDCODED_SECRET_DISABLED (479.042 µs) : 458, 500
.   : milestone, 479,
iast_INACTIVE (455.16 µs) : 433, 477
.   : milestone, 455,
iast_TELEMETRY_OFF (469.187 µs) : 448, 491
.   : milestone, 469,
tracing (443.07 µs) : 422, 464
.   : milestone, 443,
section candidate
no_agent (366.537 µs) : 345, 388
.   : milestone, 367,
iast (478.615 µs) : 458, 500
.   : milestone, 479,
iast_FULL (550.349 µs) : 528, 573
.   : milestone, 550,
iast_GLOBAL (503.107 µs) : 480, 526
.   : milestone, 503,
iast_HARDCODED_SECRET_DISABLED (477.403 µs) : 456, 498
.   : milestone, 477,
iast_INACTIVE (453.412 µs) : 431, 475
.   : milestone, 453,
iast_TELEMETRY_OFF (470.292 µs) : 449, 492
.   : milestone, 470,
tracing (439.078 µs) : 419, 459
.   : milestone, 439,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 372.444 µs [352.434 µs, 392.453 µs] -
iast 480.659 µs [459.25 µs, 502.068 µs] 108.215 µs (29.1%)
iast_FULL 547.126 µs [524.863 µs, 569.389 µs] 174.682 µs (46.9%)
iast_GLOBAL 502.352 µs [479.664 µs, 525.04 µs] 129.908 µs (34.9%)
iast_HARDCODED_SECRET_DISABLED 479.042 µs [458.233 µs, 499.851 µs] 106.598 µs (28.6%)
iast_INACTIVE 455.16 µs [432.991 µs, 477.328 µs] 82.716 µs (22.2%)
iast_TELEMETRY_OFF 469.187 µs [447.623 µs, 490.751 µs] 96.743 µs (26.0%)
tracing 443.07 µs [422.247 µs, 463.894 µs] 70.626 µs (19.0%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 366.537 µs [345.379 µs, 387.695 µs] -
iast 478.615 µs [457.577 µs, 499.654 µs] 112.078 µs (30.6%)
iast_FULL 550.349 µs [527.95 µs, 572.749 µs] 183.812 µs (50.1%)
iast_GLOBAL 503.107 µs [480.396 µs, 525.819 µs] 136.57 µs (37.3%)
iast_HARDCODED_SECRET_DISABLED 477.403 µs [456.326 µs, 498.48 µs] 110.866 µs (30.2%)
iast_INACTIVE 453.412 µs [431.397 µs, 475.426 µs] 86.874 µs (23.7%)
iast_TELEMETRY_OFF 470.292 µs [449.077 µs, 491.506 µs] 103.754 µs (28.3%)
tracing 439.078 µs [418.86 µs, 459.296 µs] 72.541 µs (19.8%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/git-changes-test
git_commit_date 1724168948 1724170688
git_commit_sha 63ccd4c a89f547
release_version 1.39.0-SNAPSHOT~63ccd4c8fc 1.39.0-SNAPSHOT~a89f5471dc
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1724172677 1724172677
ci_job_id 610844149 610844149
ci_pipeline_id 42295767 42295767
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.39.0-SNAPSHOT~a89f5471dc, baseline=1.39.0-SNAPSHOT~63ccd4c8fc
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.454 ms) : 1442, 1465
.   : milestone, 1454,
appsec (2.232 ms) : 2196, 2268
.   : milestone, 2232,
iast (1.957 ms) : 1915, 1999
.   : milestone, 1957,
iast_GLOBAL (2.022 ms) : 1978, 2066
.   : milestone, 2022,
profiling (1.87 ms) : 1834, 1906
.   : milestone, 1870,
tracing (1.833 ms) : 1800, 1866
.   : milestone, 1833,
section candidate
no_agent (1.459 ms) : 1447, 1470
.   : milestone, 1459,
appsec (2.223 ms) : 2187, 2258
.   : milestone, 2223,
iast (1.982 ms) : 1939, 2025
.   : milestone, 1982,
iast_GLOBAL (2.019 ms) : 1976, 2062
.   : milestone, 2019,
profiling (1.872 ms) : 1837, 1907
.   : milestone, 1872,
tracing (1.845 ms) : 1811, 1878
.   : milestone, 1845,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.454 ms [1.442 ms, 1.465 ms] -
appsec 2.232 ms [2.196 ms, 2.268 ms] 778.239 µs (53.5%)
iast 1.957 ms [1.915 ms, 1.999 ms] 503.099 µs (34.6%)
iast_GLOBAL 2.022 ms [1.978 ms, 2.066 ms] 568.359 µs (39.1%)
profiling 1.87 ms [1.834 ms, 1.906 ms] 416.549 µs (28.7%)
tracing 1.833 ms [1.8 ms, 1.866 ms] 379.519 µs (26.1%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.459 ms [1.447 ms, 1.47 ms] -
appsec 2.223 ms [2.187 ms, 2.258 ms] 763.859 µs (52.4%)
iast 1.982 ms [1.939 ms, 2.025 ms] 523.35 µs (35.9%)
iast_GLOBAL 2.019 ms [1.976 ms, 2.062 ms] 560.328 µs (38.4%)
profiling 1.872 ms [1.837 ms, 1.907 ms] 413.209 µs (28.3%)
tracing 1.845 ms [1.811 ms, 1.878 ms] 385.784 µs (26.4%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.39.0-SNAPSHOT~a89f5471dc, baseline=1.39.0-SNAPSHOT~63ccd4c8fc
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.012 s) : 15012000, 15012000
.   : milestone, 15012000,
appsec (15.392 s) : 15392000, 15392000
.   : milestone, 15392000,
iast (18.798 s) : 18798000, 18798000
.   : milestone, 18798000,
iast_GLOBAL (17.93 s) : 17930000, 17930000
.   : milestone, 17930000,
profiling (14.887 s) : 14887000, 14887000
.   : milestone, 14887000,
tracing (15.285 s) : 15285000, 15285000
.   : milestone, 15285000,
section candidate
no_agent (15.341 s) : 15341000, 15341000
.   : milestone, 15341000,
appsec (14.945 s) : 14945000, 14945000
.   : milestone, 14945000,
iast (18.757 s) : 18757000, 18757000
.   : milestone, 18757000,
iast_GLOBAL (18.01 s) : 18010000, 18010000
.   : milestone, 18010000,
profiling (15.075 s) : 15075000, 15075000
.   : milestone, 15075000,
tracing (15.088 s) : 15088000, 15088000
.   : milestone, 15088000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.012 s [15.012 s, 15.012 s] -
appsec 15.392 s [15.392 s, 15.392 s] 380.0 ms (2.5%)
iast 18.798 s [18.798 s, 18.798 s] 3.786 s (25.2%)
iast_GLOBAL 17.93 s [17.93 s, 17.93 s] 2.918 s (19.4%)
profiling 14.887 s [14.887 s, 14.887 s] -125.0 ms (-0.8%)
tracing 15.285 s [15.285 s, 15.285 s] 273.0 ms (1.8%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.341 s [15.341 s, 15.341 s] -
appsec 14.945 s [14.945 s, 14.945 s] -396.0 ms (-2.6%)
iast 18.757 s [18.757 s, 18.757 s] 3.416 s (22.3%)
iast_GLOBAL 18.01 s [18.01 s, 18.01 s] 2.669 s (17.4%)
profiling 15.075 s [15.075 s, 15.075 s] -266.0 ms (-1.7%)
tracing 15.088 s [15.088 s, 15.088 s] -253.0 ms (-1.6%)

@smola smola force-pushed the smola/git-changes-test branch from def6068 to 2a741cd Compare August 18, 2024 17:50
@smola smola mentioned this pull request Aug 18, 2024
Closed
@smola smola added tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling labels Aug 18, 2024
@smola smola force-pushed the smola/git-changes-test branch 2 times, most recently from 096fc38 to cc47be1 Compare August 19, 2024 07:18
@smola smola changed the base branch from master to smola/fix-ci-test-selection August 19, 2024 07:18
@pr-commenter
Copy link

pr-commenter bot commented Aug 19, 2024 •
edited
Loading

Debugger benchmarks

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
ci_job_date 1724070516 1724070896
end_time 2024-08-19T12:29:51 2024-08-19T12:36:10
git_branch master smola/git-changes-test
git_commit_sha 0c5f7c2 f8991a1
start_time 2024-08-19T12:28:37 2024-08-19T12:34:57
See matching parameters
Baseline Candidate
ci_job_id 608691802 608691802
ci_pipeline_id 42109383 42109383
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date 1724069838 1724069838

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 6 unstable metrics.

See unchanged results
scenario Δ mean agg_http_req_duration_min Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p75 Δ mean agg_http_req_duration_p99 Δ mean throughput
scenario:noprobe unstable
[-32.265µs; +18.502µs] or [-12.187%; +6.989%]		 unstable
[-44.645µs; +33.193µs] or [-14.840%; +11.034%]		 unstable
[-56.997µs; +46.152µs] or [-18.132%; +14.682%]		 unstable
[-76.785µs; +111.429µs] or [-12.144%; +17.624%]		 same
scenario:basic unsure
[-9.407µs; -0.172µs] or [-3.449%; -0.063%]		 same same unstable
[-51.339µs; +35.046µs] or [-8.534%; +5.825%]		 unstable
[-79.177op/s; +214.129op/s] or [-3.088%; +8.351%]
scenario:loop unsure
[-45.130µs; -30.743µs] or [-0.434%; -0.296%]		 unsure
[-68.471µs; -11.816µs] or [-0.648%; -0.112%]		 same same same
Request duration reports for reports 
gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (300.837 µs) : 263, 339
.   : milestone, 301,
basic (299.216 µs) : 291, 307
.   : milestone, 299,
loop (10.561 ms) : 10533, 10589
.   : milestone, 10561,
section candidate
noprobe (295.111 µs) : 259, 332
.   : milestone, 295,
basic (295.047 µs) : 286, 304
.   : milestone, 295,
loop (10.521 ms) : 10496, 10545
.   : milestone, 10521,
Loading
  • baseline results
Scenario Request median duration [CI 0.99]
noprobe 300.837 µs [262.99 µs, 338.683 µs]
basic 299.216 µs [291.197 µs, 307.234 µs]
loop 10.561 ms [10.533 ms, 10.589 ms]
  • candidate results
Scenario Request median duration [CI 0.99]
noprobe 295.111 µs [258.712 µs, 331.51 µs]
basic 295.047 µs [286.387 µs, 303.707 µs]
loop 10.521 ms [10.496 ms, 10.545 ms]

@smola smola force-pushed the smola/fix-ci-test-selection branch from cda55d5 to e2947e2 Compare August 19, 2024 10:56
@smola smola force-pushed the smola/git-changes-test branch 2 times, most recently from 87b138c to d664538 Compare August 19, 2024 11:07
Base automatically changed from smola/fix-ci-test-selection to master August 19, 2024 13:33
@smola smola force-pushed the smola/git-changes-test branch from f8991a1 to c7f43cb Compare August 19, 2024 13:34
@smola smola marked this pull request as ready for review August 19, 2024 13:45
@smola smola requested a review from a team as a code owner August 19, 2024 13:45
@smola smola requested review from PerfectSlayer and mcculls August 19, 2024 13:45
PerfectSlayer
PerfectSlayer approved these changes Aug 20, 2024
View reviewed changes
Copy link
Contributor

@PerfectSlayer PerfectSlayer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That looks nice!
I only reviewed it "on paper" (did not run it).

I am curious about if we should shadow this change, like running the change detection while still running everything and keeping a close eye to its output, or just deploy it and refine it as we go? -- I expect there will be few tasks that might not be caught from the first try 😇

@PerfectSlayer
Copy link
Contributor

Muzzle failure? 🤔

:dd-java-agent:instrumentation:couchbase:couchbase-3.2:muzzle-AssertPass-com.couchbase.client-java-client-3.6.3

It might come from master though...

@smola smola force-pushed the smola/git-changes-test branch from a7047ea to 31efcdf Compare August 20, 2024 14:32
@smola smola force-pushed the smola/git-changes-test branch from 31efcdf to a89f547 Compare August 20, 2024 16:18
@smola
Copy link
Member Author

smola commented Aug 20, 2024 •
edited
Loading

@PerfectSlayer I am curious about if we should shadow this change, like running the change detection while still running everything and keeping a close eye to its output, or just deploy it and refine it as we go?

Adding some sort of dry run sounds good in theory, the problem is that we'll probably be looking at results and think "ok, looks fine", because if it goes wrong, it'll probably be something that catches us off guard and not be obvious until we really skip it and it fails in master 🤔

@smola smola merged commit ce8ac44 into master Aug 21, 2024
@smola smola deleted the smola/git-changes-test branch August 21, 2024 13:34
@github-actions github-actions bot added this to the 1.39.0 milestone Aug 21, 2024
@smola smola mentioned this pull request Jun 26, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@PerfectSlayer PerfectSlayer PerfectSlayer approved these changes

@mcculls mcculls Awaiting requested review from mcculls mcculls is a code owner automatically assigned from DataDog/apm-java

Assignees

No one assigned

Labels

comp: tooling Build & Tooling tag: no release notes Changes to exclude from release notes

Projects

None yet

Milestone

1.39.0

Development

Successfully merging this pull request may close these issues.

2 participants

@smola @PerfectSlayer