Skip to content

TESTING - DO NOT MERGE #8827

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

TESTING - DO NOT MERGE #8827

wants to merge 1 commit into from

Conversation

mcculls
Copy link
Contributor

@mcculls mcculls commented May 14, 2025

No description provided.

@mcculls mcculls added the tag: do not merge Do not merge changes label May 14, 2025
@mcculls mcculls force-pushed the mcculls/circleci branch from 4b0e508 to e0469b8 Compare May 14, 2025 19:19
@pr-commenter
Copy link

pr-commenter bot commented May 14, 2025 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mcculls/circleci
git_commit_date 1747229243 1747251398
git_commit_sha 7c80dbe 3e546c1
release_version 1.50.0-SNAPSHOT~7c80dbede3 1.50.0-SNAPSHOT~3e546c1d55
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1747253718 1747253718
ci_job_id 938840497 938840497
ci_pipeline_id 65127853 65127853
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-5esuzn9-project-304-concurrent-0-tw999282 6.8.0-1028-aws #30~22.04.1-Ubuntu SMP Sun Apr 20 06:03:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-5esuzn9-project-304-concurrent-0-tw999282 6.8.0-1028-aws #30~22.04.1-Ubuntu SMP Sun Apr 20 06:03:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 57 metrics, 14 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~3e546c1d55, baseline=1.50.0-SNAPSHOT~7c80dbede3

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.021 s) : 0, 1020679
Total [baseline] (8.643 s) : 0, 8642739
Agent [candidate] (1.017 s) : 0, 1017296
Total [candidate] (8.631 s) : 0, 8631241
section iast
Agent [baseline] (1.15 s) : 0, 1150394
Total [baseline] (9.187 s) : 0, 9187483
Agent [candidate] (1.153 s) : 0, 1153224
Total [candidate] (9.261 s) : 0, 9261040
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.157 s) : 0, 1157256
Total [baseline] (9.238 s) : 0, 9238408
Agent [candidate] (1.15 s) : 0, 1150249
Total [candidate] (9.176 s) : 0, 9175879
section iast_TELEMETRY_OFF
Agent [baseline] (1.144 s) : 0, 1143847
Total [baseline] (9.212 s) : 0, 9211835
Agent [candidate] (1.143 s) : 0, 1142820
Total [candidate] (9.199 s) : 0, 9198835
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.021 s -
Agent iast 1.15 s 129.716 ms (12.7%)
Agent iast_HARDCODED_SECRET_DISABLED 1.157 s 136.577 ms (13.4%)
Agent iast_TELEMETRY_OFF 1.144 s 123.168 ms (12.1%)
Total tracing 8.643 s -
Total iast 9.187 s 544.744 ms (6.3%)
Total iast_HARDCODED_SECRET_DISABLED 9.238 s 595.669 ms (6.9%)
Total iast_TELEMETRY_OFF 9.212 s 569.096 ms (6.6%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.017 s -
Agent iast 1.153 s 135.928 ms (13.4%)
Agent iast_HARDCODED_SECRET_DISABLED 1.15 s 132.953 ms (13.1%)
Agent iast_TELEMETRY_OFF 1.143 s 125.524 ms (12.3%)
Total tracing 8.631 s -
Total iast 9.261 s 629.799 ms (7.3%)
Total iast_HARDCODED_SECRET_DISABLED 9.176 s 544.638 ms (6.3%)
Total iast_TELEMETRY_OFF 9.199 s 567.594 ms (6.6%) 
gantt
    title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~3e546c1d55, baseline=1.50.0-SNAPSHOT~7c80dbede3

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (681.988 ms) : 0, 681988
BytebuddyAgent [candidate] (681.372 ms) : 0, 681372
GlobalTracer [baseline] (239.883 ms) : 0, 239883
GlobalTracer [candidate] (239.463 ms) : 0, 239463
AppSec [baseline] (55.102 ms) : 0, 55102
AppSec [candidate] (54.208 ms) : 0, 54208
Debugger [baseline] (8.991 ms) : 0, 8991
Debugger [candidate] (8.974 ms) : 0, 8974
Remote Config [baseline] (697.53 µs) : 0, 698
Remote Config [candidate] (685.184 µs) : 0, 685
Telemetry [baseline] (10.571 ms) : 0, 10571
Telemetry [candidate] (9.048 ms) : 0, 9048
section iast
BytebuddyAgent [baseline] (802.493 ms) : 0, 802493
BytebuddyAgent [candidate] (803.491 ms) : 0, 803491
GlobalTracer [baseline] (230.997 ms) : 0, 230997
GlobalTracer [candidate] (232.047 ms) : 0, 232047
IAST [baseline] (26.057 ms) : 0, 26057
IAST [candidate] (27.718 ms) : 0, 27718
AppSec [baseline] (52.931 ms) : 0, 52931
AppSec [candidate] (51.803 ms) : 0, 51803
Debugger [baseline] (5.958 ms) : 0, 5958
Debugger [candidate] (5.993 ms) : 0, 5993
Remote Config [baseline] (604.247 µs) : 0, 604
Remote Config [candidate] (605.082 µs) : 0, 605
Telemetry [baseline] (7.819 ms) : 0, 7819
Telemetry [candidate] (7.994 ms) : 0, 7994
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (807.32 ms) : 0, 807320
BytebuddyAgent [candidate] (802.244 ms) : 0, 802244
GlobalTracer [baseline] (232.497 ms) : 0, 232497
GlobalTracer [candidate] (231.229 ms) : 0, 231229
IAST [baseline] (28.086 ms) : 0, 28086
IAST [candidate] (29.412 ms) : 0, 29412
AppSec [baseline] (51.264 ms) : 0, 51264
AppSec [candidate] (49.54 ms) : 0, 49540
Debugger [baseline] (5.903 ms) : 0, 5903
Debugger [candidate] (5.911 ms) : 0, 5911
Remote Config [baseline] (605.228 µs) : 0, 605
Remote Config [candidate] (601.381 µs) : 0, 601
Telemetry [baseline] (7.919 ms) : 0, 7919
Telemetry [candidate] (7.893 ms) : 0, 7893
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (798.32 ms) : 0, 798320
BytebuddyAgent [candidate] (796.268 ms) : 0, 796268
GlobalTracer [baseline] (230.225 ms) : 0, 230225
GlobalTracer [candidate] (230.52 ms) : 0, 230520
IAST [baseline] (23.006 ms) : 0, 23006
IAST [candidate] (22.199 ms) : 0, 22199
AppSec [baseline] (54.571 ms) : 0, 54571
AppSec [candidate] (56.163 ms) : 0, 56163
Debugger [baseline] (5.928 ms) : 0, 5928
Debugger [candidate] (5.947 ms) : 0, 5947
Remote Config [baseline] (593.75 µs) : 0, 594
Remote Config [candidate] (589.77 µs) : 0, 590
Telemetry [baseline] (7.75 ms) : 0, 7750
Telemetry [candidate] (7.724 ms) : 0, 7724
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~3e546c1d55, baseline=1.50.0-SNAPSHOT~7c80dbede3

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.018 s) : 0, 1018382
Total [baseline] (10.508 s) : 0, 10507628
Agent [candidate] (1.034 s) : 0, 1033710
Total [candidate] (10.518 s) : 0, 10518487
section appsec
Agent [baseline] (1.159 s) : 0, 1159226
Total [baseline] (10.664 s) : 0, 10664337
Agent [candidate] (1.167 s) : 0, 1166764
Total [candidate] (10.698 s) : 0, 10697598
section iast
Agent [baseline] (1.155 s) : 0, 1155454
Total [baseline] (10.895 s) : 0, 10894792
Agent [candidate] (1.151 s) : 0, 1150951
Total [candidate] (10.937 s) : 0, 10937194
section profiling
Agent [baseline] (1.278 s) : 0, 1278429
Total [baseline] (10.795 s) : 0, 10794894
Agent [candidate] (1.288 s) : 0, 1287590
Total [candidate] (10.892 s) : 0, 10891980
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.018 s -
Agent appsec 1.159 s 140.844 ms (13.8%)
Agent iast 1.155 s 137.072 ms (13.5%)
Agent profiling 1.278 s 260.047 ms (25.5%)
Total tracing 10.508 s -
Total appsec 10.664 s 156.709 ms (1.5%)
Total iast 10.895 s 387.164 ms (3.7%)
Total profiling 10.795 s 287.266 ms (2.7%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.034 s -
Agent appsec 1.167 s 133.054 ms (12.9%)
Agent iast 1.151 s 117.241 ms (11.3%)
Agent profiling 1.288 s 253.88 ms (24.6%)
Total tracing 10.518 s -
Total appsec 10.698 s 179.111 ms (1.7%)
Total iast 10.937 s 418.707 ms (4.0%)
Total profiling 10.892 s 373.492 ms (3.6%) 
gantt
    title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~3e546c1d55, baseline=1.50.0-SNAPSHOT~7c80dbede3

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (682.11 ms) : 0, 682110
BytebuddyAgent [candidate] (692.184 ms) : 0, 692184
GlobalTracer [baseline] (240.382 ms) : 0, 240382
GlobalTracer [candidate] (243.547 ms) : 0, 243547
AppSec [baseline] (54.349 ms) : 0, 54349
AppSec [candidate] (55.989 ms) : 0, 55989
Debugger [baseline] (8.282 ms) : 0, 8282
Debugger [candidate] (9.002 ms) : 0, 9002
Remote Config [baseline] (712.246 µs) : 0, 712
Remote Config [candidate] (701.636 µs) : 0, 702
Telemetry [baseline] (9.047 ms) : 0, 9047
Telemetry [candidate] (8.443 ms) : 0, 8443
section appsec
BytebuddyAgent [baseline] (699.0 ms) : 0, 699000
BytebuddyAgent [candidate] (704.561 ms) : 0, 704561
GlobalTracer [baseline] (236.42 ms) : 0, 236420
GlobalTracer [candidate] (237.828 ms) : 0, 237828
AppSec [baseline] (174.809 ms) : 0, 174809
AppSec [candidate] (175.717 ms) : 0, 175717
Debugger [baseline] (6.356 ms) : 0, 6356
Debugger [candidate] (5.975 ms) : 0, 5975
Remote Config [baseline] (618.413 µs) : 0, 618
Remote Config [candidate] (627.933 µs) : 0, 628
Telemetry [baseline] (7.727 ms) : 0, 7727
Telemetry [candidate] (7.435 ms) : 0, 7435
IAST [baseline] (21.615 ms) : 0, 21615
IAST [candidate] (21.92 ms) : 0, 21920
section iast
BytebuddyAgent [baseline] (807.428 ms) : 0, 807428
BytebuddyAgent [candidate] (803.208 ms) : 0, 803208
GlobalTracer [baseline] (231.031 ms) : 0, 231031
GlobalTracer [candidate] (231.172 ms) : 0, 231172
AppSec [baseline] (52.088 ms) : 0, 52088
AppSec [candidate] (48.549 ms) : 0, 48549
Debugger [baseline] (5.926 ms) : 0, 5926
Debugger [candidate] (5.896 ms) : 0, 5896
Remote Config [baseline] (586.882 µs) : 0, 587
Remote Config [candidate] (587.637 µs) : 0, 588
Telemetry [baseline] (7.854 ms) : 0, 7854
Telemetry [candidate] (7.841 ms) : 0, 7841
IAST [baseline] (26.937 ms) : 0, 26937
IAST [candidate] (30.226 ms) : 0, 30226
section profiling
ProfilingAgent [baseline] (102.94 ms) : 0, 102940
ProfilingAgent [candidate] (104.633 ms) : 0, 104633
BytebuddyAgent [baseline] (672.892 ms) : 0, 672892
BytebuddyAgent [candidate] (677.061 ms) : 0, 677061
GlobalTracer [baseline] (374.695 ms) : 0, 374695
GlobalTracer [candidate] (377.033 ms) : 0, 377033
AppSec [baseline] (62.232 ms) : 0, 62232
AppSec [candidate] (62.433 ms) : 0, 62433
Debugger [baseline] (6.248 ms) : 0, 6248
Debugger [candidate] (6.34 ms) : 0, 6340
Remote Config [baseline] (656.048 µs) : 0, 656
Remote Config [candidate] (669.441 µs) : 0, 669
Telemetry [baseline] (8.159 ms) : 0, 8159
Telemetry [candidate] (8.407 ms) : 0, 8407
Profiling [baseline] (102.965 ms) : 0, 102965
Profiling [candidate] (104.657 ms) : 0, 104657
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2025-05-14T19:45:59 2025-05-14T19:53:43
git_branch master mcculls/circleci
git_commit_date 1747229243 1747251398
git_commit_sha 7c80dbe 3e546c1
release_version 1.50.0-SNAPSHOT~7c80dbede3 1.50.0-SNAPSHOT~3e546c1d55
start_time 2025-05-14T19:45:45 2025-05-14T19:53:28
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1747252822 1747252822
ci_job_id 938840498 938840498
ci_pipeline_id 65127853 65127853
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-5esuzn9-project-304-concurrent-1-zimh2xpb 6.8.0-1028-aws #30~22.04.1-Ubuntu SMP Sun Apr 20 06:03:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-5esuzn9-project-304-concurrent-1-zimh2xpb 6.8.0-1028-aws #30~22.04.1-Ubuntu SMP Sun Apr 20 06:03:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant iast iast

Summary

Found 0 performance improvements and 1 performance regressions! Performance is the same for 11 metrics, 18 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:petclinic:profiling worse
[+50.953µs; +103.893µs] or [+3.411%; +6.956%]		 unstable
[-681.706op/s; +453.786op/s] or [-22.155%; +14.748%]		 1.571ms 2962.963op/s 1.494ms 3076.923op/s
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~3e546c1d55, baseline=1.50.0-SNAPSHOT~7c80dbede3
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.384 ms) : 1364, 1404
.   : milestone, 1384,
appsec (1.74 ms) : 1717, 1763
.   : milestone, 1740,
appsec_no_iast (1.727 ms) : 1704, 1750
.   : milestone, 1727,
code_origins (1.678 ms) : 1648, 1708
.   : milestone, 1678,
iast (1.537 ms) : 1510, 1563
.   : milestone, 1537,
profiling (1.494 ms) : 1470, 1517
.   : milestone, 1494,
tracing (1.501 ms) : 1476, 1526
.   : milestone, 1501,
section candidate
no_agent (1.377 ms) : 1357, 1397
.   : milestone, 1377,
appsec (1.747 ms) : 1724, 1771
.   : milestone, 1747,
appsec_no_iast (1.734 ms) : 1711, 1757
.   : milestone, 1734,
code_origins (1.679 ms) : 1652, 1706
.   : milestone, 1679,
iast (1.527 ms) : 1504, 1551
.   : milestone, 1527,
profiling (1.571 ms) : 1545, 1597
.   : milestone, 1571,
tracing (1.494 ms) : 1470, 1518
.   : milestone, 1494,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.384 ms [1.364 ms, 1.404 ms] -
appsec 1.74 ms [1.717 ms, 1.763 ms] 356.259 µs (25.7%)
appsec_no_iast 1.727 ms [1.704 ms, 1.75 ms] 343.406 µs (24.8%)
code_origins 1.678 ms [1.648 ms, 1.708 ms] 294.558 µs (21.3%)
iast 1.537 ms [1.51 ms, 1.563 ms] 152.919 µs (11.1%)
profiling 1.494 ms [1.47 ms, 1.517 ms] 109.977 µs (7.9%)
tracing 1.501 ms [1.476 ms, 1.526 ms] 117.656 µs (8.5%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.377 ms [1.357 ms, 1.397 ms] -
appsec 1.747 ms [1.724 ms, 1.771 ms] 370.652 µs (26.9%)
appsec_no_iast 1.734 ms [1.711 ms, 1.757 ms] 357.288 µs (26.0%)
code_origins 1.679 ms [1.652 ms, 1.706 ms] 302.34 µs (22.0%)
iast 1.527 ms [1.504 ms, 1.551 ms] 150.573 µs (10.9%)
profiling 1.571 ms [1.545 ms, 1.597 ms] 194.312 µs (14.1%)
tracing 1.494 ms [1.47 ms, 1.518 ms] 117.41 µs (8.5%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~3e546c1d55, baseline=1.50.0-SNAPSHOT~7c80dbede3
    dateFormat X
    axisFormat %s
section baseline
no_agent (378.431 µs) : 358, 399
.   : milestone, 378,
iast (518.262 µs) : 496, 540
.   : milestone, 518,
iast_FULL (738.027 µs) : 716, 760
.   : milestone, 738,
iast_GLOBAL (560.753 µs) : 539, 582
.   : milestone, 561,
iast_HARDCODED_SECRET_DISABLED (526.967 µs) : 504, 550
.   : milestone, 527,
iast_INACTIVE (465.706 µs) : 443, 488
.   : milestone, 466,
iast_TELEMETRY_OFF (503.886 µs) : 481, 527
.   : milestone, 504,
tracing (468.832 µs) : 446, 491
.   : milestone, 469,
section candidate
no_agent (384.446 µs) : 365, 404
.   : milestone, 384,
iast (517.041 µs) : 495, 539
.   : milestone, 517,
iast_FULL (740.203 µs) : 718, 762
.   : milestone, 740,
iast_GLOBAL (564.891 µs) : 542, 588
.   : milestone, 565,
iast_HARDCODED_SECRET_DISABLED (522.884 µs) : 500, 546
.   : milestone, 523,
iast_INACTIVE (466.124 µs) : 444, 488
.   : milestone, 466,
iast_TELEMETRY_OFF (515.242 µs) : 492, 539
.   : milestone, 515,
tracing (464.441 µs) : 442, 487
.   : milestone, 464,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 378.431 µs [358.313 µs, 398.549 µs] -
iast 518.262 µs [496.059 µs, 540.465 µs] 139.831 µs (37.0%)
iast_FULL 738.027 µs [716.006 µs, 760.048 µs] 359.596 µs (95.0%)
iast_GLOBAL 560.753 µs [539.139 µs, 582.366 µs] 182.322 µs (48.2%)
iast_HARDCODED_SECRET_DISABLED 526.967 µs [504.362 µs, 549.573 µs] 148.536 µs (39.3%)
iast_INACTIVE 465.706 µs [443.088 µs, 488.324 µs] 87.275 µs (23.1%)
iast_TELEMETRY_OFF 503.886 µs [480.808 µs, 526.964 µs] 125.454 µs (33.2%)
tracing 468.832 µs [446.228 µs, 491.436 µs] 90.401 µs (23.9%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 384.446 µs [365.015 µs, 403.876 µs] -
iast 517.041 µs [495.164 µs, 538.918 µs] 132.596 µs (34.5%)
iast_FULL 740.203 µs [718.093 µs, 762.313 µs] 355.757 µs (92.5%)
iast_GLOBAL 564.891 µs [542.166 µs, 587.616 µs] 180.445 µs (46.9%)
iast_HARDCODED_SECRET_DISABLED 522.884 µs [499.916 µs, 545.853 µs] 138.439 µs (36.0%)
iast_INACTIVE 466.124 µs [444.245 µs, 488.002 µs] 81.678 µs (21.2%)
iast_TELEMETRY_OFF 515.242 µs [491.81 µs, 538.674 µs] 130.797 µs (34.0%)
tracing 464.441 µs [442.136 µs, 486.745 µs] 79.995 µs (20.8%)

Dacapo

@mcculls mcculls force-pushed the mcculls/circleci branch from e0469b8 to 3e546c1 Compare May 14, 2025 19:36
@mcculls mcculls force-pushed the mcculls/circleci branch from 3e546c1 to 78704f5 Compare May 14, 2025 20:09
@mcculls mcculls closed this May 14, 2025
@mcculls mcculls deleted the mcculls/circleci branch May 14, 2025 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

tag: do not merge Do not merge changes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@mcculls