Skip to content

Conversation

nikita-tkachenko-datadog
Copy link
Contributor

@nikita-tkachenko-datadog nikita-tkachenko-datadog commented Jan 27, 2025

What Does This Do

Does some updates to TestRetryPolicy and the code that uses it:

  • retry reason is now an enum instead of a String
  • the contract of suppressFailures() method is changed to return whether failures should be suppressed for the current execution
  • currentExecutionIsRetry() method is removed

Motivation

Preparing for future flaky tests management changes: quarantined tests will have to be executed without affecting the status of the build.

Contributor Checklist

Jira ticket: SDTEST-1483

@pr-commenter
Copy link

pr-commenter bot commented Jan 27, 2025

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nikita-tkachenko/generalize-test-retries
git_commit_date 1738137590 1738143458
git_commit_sha f06d59c 5f23339
release_version 1.46.0-SNAPSHOT~f06d59c0dd 1.46.0-SNAPSHOT~5f23339385
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1738145929 1738145929
ci_job_id 784984256 784984256
ci_pipeline_id 54289730 54289730
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-qirit7hx-project-304-concurrent-0-yybczn9i 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Linux runner-qirit7hx-project-304-concurrent-0-yybczn9i 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 51 metrics, 12 unstable metrics.

Startup time reports for petclinic
gantt
    title petclinic - global startup overhead: candidate=1.46.0-SNAPSHOT~5f23339385, baseline=1.46.0-SNAPSHOT~f06d59c0dd

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.045 s) : 0, 1044758
Total [baseline] (10.469 s) : 0, 10468729
Agent [candidate] (1.043 s) : 0, 1042987
Total [candidate] (10.43 s) : 0, 10430332
section appsec
Agent [baseline] (1.184 s) : 0, 1183867
Total [baseline] (10.775 s) : 0, 10774678
Agent [candidate] (1.186 s) : 0, 1186078
Total [candidate] (10.796 s) : 0, 10796442
section iast
Agent [baseline] (1.172 s) : 0, 1172343
Total [baseline] (10.968 s) : 0, 10968202
Agent [candidate] (1.172 s) : 0, 1172489
Total [candidate] (10.925 s) : 0, 10924630
section profiling
Agent [baseline] (1.265 s) : 0, 1265248
Total [baseline] (10.877 s) : 0, 10877376
Agent [candidate] (1.267 s) : 0, 1267410
Total [candidate] (10.954 s) : 0, 10954253
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.045 s -
Agent appsec 1.184 s 139.109 ms (13.3%)
Agent iast 1.172 s 127.584 ms (12.2%)
Agent profiling 1.265 s 220.49 ms (21.1%)
Total tracing 10.469 s -
Total appsec 10.775 s 305.949 ms (2.9%)
Total iast 10.968 s 499.473 ms (4.8%)
Total profiling 10.877 s 408.647 ms (3.9%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.043 s -
Agent appsec 1.186 s 143.091 ms (13.7%)
Agent iast 1.172 s 129.503 ms (12.4%)
Agent profiling 1.267 s 224.423 ms (21.5%)
Total tracing 10.43 s -
Total appsec 10.796 s 366.111 ms (3.5%)
Total iast 10.925 s 494.299 ms (4.7%)
Total profiling 10.954 s 523.921 ms (5.0%)
gantt
    title petclinic - break down per module: candidate=1.46.0-SNAPSHOT~5f23339385, baseline=1.46.0-SNAPSHOT~f06d59c0dd

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (719.31 ms) : 0, 719310
BytebuddyAgent [candidate] (716.64 ms) : 0, 716640
GlobalTracer [baseline] (242.099 ms) : 0, 242099
GlobalTracer [candidate] (240.673 ms) : 0, 240673
AppSec [baseline] (55.851 ms) : 0, 55851
AppSec [candidate] (56.219 ms) : 0, 56219
Remote Config [baseline] (719.823 µs) : 0, 720
Remote Config [candidate] (709.007 µs) : 0, 709
Telemetry [baseline] (11.525 ms) : 0, 11525
Telemetry [candidate] (13.48 ms) : 0, 13480
section appsec
BytebuddyAgent [baseline] (734.64 ms) : 0, 734640
BytebuddyAgent [candidate] (735.249 ms) : 0, 735249
GlobalTracer [baseline] (237.684 ms) : 0, 237684
GlobalTracer [candidate] (238.646 ms) : 0, 238646
AppSec [baseline] (176.593 ms) : 0, 176593
AppSec [candidate] (176.985 ms) : 0, 176985
Remote Config [baseline] (651.915 µs) : 0, 652
Remote Config [candidate] (668.067 µs) : 0, 668
Telemetry [baseline] (8.285 ms) : 0, 8285
Telemetry [candidate] (8.352 ms) : 0, 8352
IAST [baseline] (21.34 ms) : 0, 21340
IAST [candidate] (21.653 ms) : 0, 21653
section iast
BytebuddyAgent [baseline] (836.032 ms) : 0, 836032
BytebuddyAgent [candidate] (836.486 ms) : 0, 836486
GlobalTracer [baseline] (231.685 ms) : 0, 231685
GlobalTracer [candidate] (231.639 ms) : 0, 231639
AppSec [baseline] (54.014 ms) : 0, 54014
AppSec [candidate] (55.543 ms) : 0, 55543
Remote Config [baseline] (618.356 µs) : 0, 618
Remote Config [candidate] (619.137 µs) : 0, 619
Telemetry [baseline] (8.852 ms) : 0, 8852
Telemetry [candidate] (8.773 ms) : 0, 8773
IAST [baseline] (25.798 ms) : 0, 25798
IAST [candidate] (24.056 ms) : 0, 24056
section profiling
BytebuddyAgent [baseline] (709.959 ms) : 0, 709959
BytebuddyAgent [candidate] (710.521 ms) : 0, 710521
GlobalTracer [baseline] (351.181 ms) : 0, 351181
GlobalTracer [candidate] (353.893 ms) : 0, 353893
AppSec [baseline] (55.488 ms) : 0, 55488
AppSec [candidate] (54.397 ms) : 0, 54397
Remote Config [baseline] (712.904 µs) : 0, 713
Remote Config [candidate] (733.68 µs) : 0, 734
Telemetry [baseline] (8.945 ms) : 0, 8945
Telemetry [candidate] (8.942 ms) : 0, 8942
ProfilingAgent [baseline] (96.368 ms) : 0, 96368
ProfilingAgent [candidate] (96.262 ms) : 0, 96262
Profiling [baseline] (96.392 ms) : 0, 96392
Profiling [candidate] (96.287 ms) : 0, 96287
Loading
Startup time reports for insecure-bank
gantt
    title insecure-bank - global startup overhead: candidate=1.46.0-SNAPSHOT~5f23339385, baseline=1.46.0-SNAPSHOT~f06d59c0dd

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.04 s) : 0, 1039995
Total [baseline] (8.67 s) : 0, 8670116
Agent [candidate] (1.039 s) : 0, 1038844
Total [candidate] (8.64 s) : 0, 8639798
section iast
Agent [baseline] (1.167 s) : 0, 1167167
Total [baseline] (9.231 s) : 0, 9230870
Agent [candidate] (1.178 s) : 0, 1177766
Total [candidate] (9.254 s) : 0, 9253990
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.17 s) : 0, 1169657
Total [baseline] (9.199 s) : 0, 9198888
Agent [candidate] (1.17 s) : 0, 1169691
Total [candidate] (9.23 s) : 0, 9230205
section iast_TELEMETRY_OFF
Agent [baseline] (1.164 s) : 0, 1164489
Total [baseline] (9.255 s) : 0, 9255253
Agent [candidate] (1.192 s) : 0, 1192356
Total [candidate] (9.325 s) : 0, 9325096
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.04 s -
Agent iast 1.167 s 127.172 ms (12.2%)
Agent iast_HARDCODED_SECRET_DISABLED 1.17 s 129.662 ms (12.5%)
Agent iast_TELEMETRY_OFF 1.164 s 124.494 ms (12.0%)
Total tracing 8.67 s -
Total iast 9.231 s 560.754 ms (6.5%)
Total iast_HARDCODED_SECRET_DISABLED 9.199 s 528.772 ms (6.1%)
Total iast_TELEMETRY_OFF 9.255 s 585.137 ms (6.7%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.039 s -
Agent iast 1.178 s 138.922 ms (13.4%)
Agent iast_HARDCODED_SECRET_DISABLED 1.17 s 130.847 ms (12.6%)
Agent iast_TELEMETRY_OFF 1.192 s 153.512 ms (14.8%)
Total tracing 8.64 s -
Total iast 9.254 s 614.192 ms (7.1%)
Total iast_HARDCODED_SECRET_DISABLED 9.23 s 590.406 ms (6.8%)
Total iast_TELEMETRY_OFF 9.325 s 685.298 ms (7.9%)
gantt
    title insecure-bank - break down per module: candidate=1.46.0-SNAPSHOT~5f23339385, baseline=1.46.0-SNAPSHOT~f06d59c0dd

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (715.186 ms) : 0, 715186
BytebuddyAgent [candidate] (714.238 ms) : 0, 714238
GlobalTracer [baseline] (240.522 ms) : 0, 240522
GlobalTracer [candidate] (239.475 ms) : 0, 239475
AppSec [baseline] (56.705 ms) : 0, 56705
AppSec [candidate] (56.278 ms) : 0, 56278
Remote Config [baseline] (729.239 µs) : 0, 729
Remote Config [candidate] (713.706 µs) : 0, 714
Telemetry [baseline] (11.641 ms) : 0, 11641
Telemetry [candidate] (12.947 ms) : 0, 12947
section iast
BytebuddyAgent [baseline] (832.916 ms) : 0, 832916
BytebuddyAgent [candidate] (840.561 ms) : 0, 840561
GlobalTracer [baseline] (230.167 ms) : 0, 230167
GlobalTracer [candidate] (231.788 ms) : 0, 231788
AppSec [baseline] (55.497 ms) : 0, 55497
AppSec [candidate] (53.807 ms) : 0, 53807
Remote Config [baseline] (625.864 µs) : 0, 626
Remote Config [candidate] (636.137 µs) : 0, 636
Telemetry [baseline] (8.73 ms) : 0, 8730
Telemetry [candidate] (8.937 ms) : 0, 8937
IAST [baseline] (23.926 ms) : 0, 23926
IAST [candidate] (26.638 ms) : 0, 26638
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (834.322 ms) : 0, 834322
BytebuddyAgent [candidate] (833.888 ms) : 0, 833888
GlobalTracer [baseline] (230.398 ms) : 0, 230398
GlobalTracer [candidate] (230.929 ms) : 0, 230929
AppSec [baseline] (54.149 ms) : 0, 54149
AppSec [candidate] (54.241 ms) : 0, 54241
Remote Config [baseline] (626.213 µs) : 0, 626
Remote Config [candidate] (625.301 µs) : 0, 625
Telemetry [baseline] (8.866 ms) : 0, 8866
Telemetry [candidate] (8.807 ms) : 0, 8807
IAST [baseline] (25.952 ms) : 0, 25952
IAST [candidate] (25.876 ms) : 0, 25876
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (830.025 ms) : 0, 830025
BytebuddyAgent [candidate] (851.234 ms) : 0, 851234
GlobalTracer [baseline] (230.693 ms) : 0, 230693
GlobalTracer [candidate] (235.063 ms) : 0, 235063
AppSec [baseline] (53.335 ms) : 0, 53335
AppSec [candidate] (55.896 ms) : 0, 55896
Remote Config [baseline] (619.719 µs) : 0, 620
Remote Config [candidate] (639.167 µs) : 0, 639
Telemetry [baseline] (8.598 ms) : 0, 8598
Telemetry [candidate] (8.792 ms) : 0, 8792
IAST [baseline] (25.885 ms) : 0, 25885
IAST [candidate] (25.088 ms) : 0, 25088
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2025-01-29T09:48:43 2025-01-29T09:55:51
git_branch master nikita-tkachenko/generalize-test-retries
git_commit_date 1738137590 1738143458
git_commit_sha f06d59c 5f23339
release_version 1.46.0-SNAPSHOT~f06d59c0dd 1.46.0-SNAPSHOT~5f23339385
start_time 2025-01-29T09:48:29 2025-01-29T09:55:37
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1738144911 1738144911
ci_job_id 784984257 784984257
ci_pipeline_id 54289730 54289730
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-k9ef9rs-project-304-concurrent-0-mrqfuojy 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Linux runner-k9ef9rs-project-304-concurrent-0-mrqfuojy 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics.

Request duration reports for petclinic
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.46.0-SNAPSHOT~5f23339385, baseline=1.46.0-SNAPSHOT~f06d59c0dd
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.368 ms) : 1348, 1388
.   : milestone, 1368,
appsec (1.766 ms) : 1743, 1789
.   : milestone, 1766,
appsec_no_iast (1.767 ms) : 1743, 1791
.   : milestone, 1767,
iast (1.518 ms) : 1494, 1542
.   : milestone, 1518,
profiling (1.57 ms) : 1545, 1595
.   : milestone, 1570,
tracing (1.486 ms) : 1461, 1512
.   : milestone, 1486,
section candidate
no_agent (1.371 ms) : 1351, 1391
.   : milestone, 1371,
appsec (1.763 ms) : 1739, 1787
.   : milestone, 1763,
appsec_no_iast (1.763 ms) : 1738, 1787
.   : milestone, 1763,
iast (1.516 ms) : 1492, 1540
.   : milestone, 1516,
profiling (1.515 ms) : 1491, 1539
.   : milestone, 1515,
tracing (1.512 ms) : 1488, 1536
.   : milestone, 1512,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.368 ms [1.348 ms, 1.388 ms] -
appsec 1.766 ms [1.743 ms, 1.789 ms] 397.799 µs (29.1%)
appsec_no_iast 1.767 ms [1.743 ms, 1.791 ms] 398.867 µs (29.2%)
iast 1.518 ms [1.494 ms, 1.542 ms] 149.472 µs (10.9%)
profiling 1.57 ms [1.545 ms, 1.595 ms] 201.999 µs (14.8%)
tracing 1.486 ms [1.461 ms, 1.512 ms] 118.319 µs (8.6%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.371 ms [1.351 ms, 1.391 ms] -
appsec 1.763 ms [1.739 ms, 1.787 ms] 391.774 µs (28.6%)
appsec_no_iast 1.763 ms [1.738 ms, 1.787 ms] 391.562 µs (28.6%)
iast 1.516 ms [1.492 ms, 1.54 ms] 144.825 µs (10.6%)
profiling 1.515 ms [1.491 ms, 1.539 ms] 144.256 µs (10.5%)
tracing 1.512 ms [1.488 ms, 1.536 ms] 140.786 µs (10.3%)
Request duration reports for insecure-bank
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.46.0-SNAPSHOT~5f23339385, baseline=1.46.0-SNAPSHOT~f06d59c0dd
    dateFormat X
    axisFormat %s
section baseline
no_agent (385.87 µs) : 366, 406
.   : milestone, 386,
iast (517.236 µs) : 496, 539
.   : milestone, 517,
iast_FULL (757.241 µs) : 734, 780
.   : milestone, 757,
iast_GLOBAL (574.313 µs) : 550, 599
.   : milestone, 574,
iast_HARDCODED_SECRET_DISABLED (517.986 µs) : 495, 540
.   : milestone, 518,
iast_INACTIVE (470.838 µs) : 449, 492
.   : milestone, 471,
iast_TELEMETRY_OFF (504.475 µs) : 481, 528
.   : milestone, 504,
tracing (463.483 µs) : 442, 485
.   : milestone, 463,
section candidate
no_agent (380.456 µs) : 361, 400
.   : milestone, 380,
iast (516.247 µs) : 493, 539
.   : milestone, 516,
iast_FULL (749.18 µs) : 726, 772
.   : milestone, 749,
iast_GLOBAL (570.577 µs) : 548, 593
.   : milestone, 571,
iast_HARDCODED_SECRET_DISABLED (518.027 µs) : 496, 540
.   : milestone, 518,
iast_INACTIVE (467.477 µs) : 446, 489
.   : milestone, 467,
iast_TELEMETRY_OFF (506.888 µs) : 483, 531
.   : milestone, 507,
tracing (467.503 µs) : 446, 489
.   : milestone, 468,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 385.87 µs [366.059 µs, 405.681 µs] -
iast 517.236 µs [495.529 µs, 538.943 µs] 131.366 µs (34.0%)
iast_FULL 757.241 µs [734.285 µs, 780.198 µs] 371.371 µs (96.2%)
iast_GLOBAL 574.313 µs [549.52 µs, 599.105 µs] 188.443 µs (48.8%)
iast_HARDCODED_SECRET_DISABLED 517.986 µs [495.473 µs, 540.498 µs] 132.116 µs (34.2%)
iast_INACTIVE 470.838 µs [449.436 µs, 492.24 µs] 84.968 µs (22.0%)
iast_TELEMETRY_OFF 504.475 µs [481.446 µs, 527.504 µs] 118.605 µs (30.7%)
tracing 463.483 µs [442.247 µs, 484.72 µs] 77.613 µs (20.1%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 380.456 µs [360.801 µs, 400.11 µs] -
iast 516.247 µs [493.045 µs, 539.449 µs] 135.791 µs (35.7%)
iast_FULL 749.18 µs [726.143 µs, 772.216 µs] 368.724 µs (96.9%)
iast_GLOBAL 570.577 µs [547.942 µs, 593.212 µs] 190.121 µs (50.0%)
iast_HARDCODED_SECRET_DISABLED 518.027 µs [495.593 µs, 540.46 µs] 137.571 µs (36.2%)
iast_INACTIVE 467.477 µs [446.233 µs, 488.722 µs] 87.021 µs (22.9%)
iast_TELEMETRY_OFF 506.888 µs [482.698 µs, 531.078 µs] 126.432 µs (33.2%)
tracing 467.503 µs [446.499 µs, 488.507 µs] 87.047 µs (22.9%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nikita-tkachenko/generalize-test-retries
git_commit_date 1738137590 1738143458
git_commit_sha f06d59c 5f23339
release_version 1.46.0-SNAPSHOT~f06d59c0dd 1.46.0-SNAPSHOT~5f23339385
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1738145462 1738145462
ci_job_id 784984258 784984258
ci_pipeline_id 54289730 54289730
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-qirit7hx-project-304-concurrent-1-ft79jxta 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Linux runner-qirit7hx-project-304-concurrent-1-ft79jxta 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for biojava
gantt
    title biojava - execution time [CI 0.99] : candidate=1.46.0-SNAPSHOT~5f23339385, baseline=1.46.0-SNAPSHOT~f06d59c0dd
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.952 s) : 14952000, 14952000
.   : milestone, 14952000,
appsec (15.063 s) : 15063000, 15063000
.   : milestone, 15063000,
iast (19.081 s) : 19081000, 19081000
.   : milestone, 19081000,
iast_GLOBAL (18.062 s) : 18062000, 18062000
.   : milestone, 18062000,
profiling (15.054 s) : 15054000, 15054000
.   : milestone, 15054000,
tracing (15.191 s) : 15191000, 15191000
.   : milestone, 15191000,
section candidate
no_agent (14.88 s) : 14880000, 14880000
.   : milestone, 14880000,
appsec (14.828 s) : 14828000, 14828000
.   : milestone, 14828000,
iast (19.094 s) : 19094000, 19094000
.   : milestone, 19094000,
iast_GLOBAL (17.926 s) : 17926000, 17926000
.   : milestone, 17926000,
profiling (15.127 s) : 15127000, 15127000
.   : milestone, 15127000,
tracing (15.17 s) : 15170000, 15170000
.   : milestone, 15170000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.952 s [14.952 s, 14.952 s] -
appsec 15.063 s [15.063 s, 15.063 s] 111.0 ms (0.7%)
iast 19.081 s [19.081 s, 19.081 s] 4.129 s (27.6%)
iast_GLOBAL 18.062 s [18.062 s, 18.062 s] 3.11 s (20.8%)
profiling 15.054 s [15.054 s, 15.054 s] 102.0 ms (0.7%)
tracing 15.191 s [15.191 s, 15.191 s] 239.0 ms (1.6%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.88 s [14.88 s, 14.88 s] -
appsec 14.828 s [14.828 s, 14.828 s] -52.0 ms (-0.3%)
iast 19.094 s [19.094 s, 19.094 s] 4.214 s (28.3%)
iast_GLOBAL 17.926 s [17.926 s, 17.926 s] 3.046 s (20.5%)
profiling 15.127 s [15.127 s, 15.127 s] 247.0 ms (1.7%)
tracing 15.17 s [15.17 s, 15.17 s] 290.0 ms (1.9%)
Execution time for tomcat
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.46.0-SNAPSHOT~5f23339385, baseline=1.46.0-SNAPSHOT~f06d59c0dd
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.474 ms) : 1463, 1486
.   : milestone, 1474,
appsec (2.361 ms) : 2318, 2404
.   : milestone, 2361,
iast (2.115 ms) : 2060, 2169
.   : milestone, 2115,
iast_GLOBAL (2.152 ms) : 2097, 2207
.   : milestone, 2152,
profiling (2.46 ms) : 2272, 2648
.   : milestone, 2460,
tracing (1.952 ms) : 1909, 1994
.   : milestone, 1952,
section candidate
no_agent (1.473 ms) : 1461, 1484
.   : milestone, 1473,
appsec (2.363 ms) : 2319, 2407
.   : milestone, 2363,
iast (2.109 ms) : 2054, 2164
.   : milestone, 2109,
iast_GLOBAL (2.15 ms) : 2095, 2205
.   : milestone, 2150,
profiling (1.977 ms) : 1934, 2021
.   : milestone, 1977,
tracing (1.945 ms) : 1903, 1987
.   : milestone, 1945,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.474 ms [1.463 ms, 1.486 ms] -
appsec 2.361 ms [2.318 ms, 2.404 ms] 886.813 µs (60.2%)
iast 2.115 ms [2.06 ms, 2.169 ms] 640.44 µs (43.4%)
iast_GLOBAL 2.152 ms [2.097 ms, 2.207 ms] 677.593 µs (46.0%)
profiling 2.46 ms [2.272 ms, 2.648 ms] 985.701 µs (66.9%)
tracing 1.952 ms [1.909 ms, 1.994 ms] 477.347 µs (32.4%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.473 ms [1.461 ms, 1.484 ms] -
appsec 2.363 ms [2.319 ms, 2.407 ms] 890.37 µs (60.5%)
iast 2.109 ms [2.054 ms, 2.164 ms] 636.276 µs (43.2%)
iast_GLOBAL 2.15 ms [2.095 ms, 2.205 ms] 677.295 µs (46.0%)
profiling 1.977 ms [1.934 ms, 2.021 ms] 504.576 µs (34.3%)
tracing 1.945 ms [1.903 ms, 1.987 ms] 471.949 µs (32.0%)

Base automatically changed from nikita-tkachenko/generalize-test-skipping to master January 28, 2025 16:47
@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/generalize-test-retries branch from 4d8593e to 3e58f24 Compare January 28, 2025 16:52
@nikita-tkachenko-datadog nikita-tkachenko-datadog marked this pull request as ready for review January 29, 2025 12:57
@nikita-tkachenko-datadog nikita-tkachenko-datadog requested a review from a team as a code owner January 29, 2025 12:57
@nikita-tkachenko-datadog nikita-tkachenko-datadog merged commit bcddb29 into master Jan 29, 2025
162 of 194 checks passed
@nikita-tkachenko-datadog nikita-tkachenko-datadog deleted the nikita-tkachenko/generalize-test-retries branch January 29, 2025 13:23
@github-actions github-actions bot added this to the 1.46.0 milestone Jan 29, 2025
svc-squareup-copybara pushed a commit to cashapp/misk that referenced this pull request Jan 31, 2025
| Package | Type | Package file | Manager | Update | Change |
|---|---|---|---|---|---|
|
[com.google.cloud:google-cloud-datastore](https://github.com/googleapis/java-datastore)
| dependencies | misk/gradle/libs.versions.toml | gradle | minor |
`2.25.4` -> `2.26.0` |
| [com.datadoghq:dd-trace-api](https://github.com/datadog/dd-trace-java)
| dependencies | misk/gradle/libs.versions.toml | gradle | minor |
`1.45.2` -> `1.46.0` |
| [com.datadoghq:dd-trace-ot](https://github.com/datadog/dd-trace-java)
| dependencies | misk/gradle/libs.versions.toml | gradle | minor |
`1.45.2` -> `1.46.0` |
| [software.amazon.awssdk:sdk-core](https://aws.amazon.com/sdkforjava) |
dependencies | misk/gradle/libs.versions.toml | gradle | patch |
`2.30.9` -> `2.30.10` |
|
[software.amazon.awssdk:dynamodb-enhanced](https://aws.amazon.com/sdkforjava)
| dependencies | misk/gradle/libs.versions.toml | gradle | patch |
`2.30.9` -> `2.30.10` |
| [software.amazon.awssdk:dynamodb](https://aws.amazon.com/sdkforjava) |
dependencies | misk/gradle/libs.versions.toml | gradle | patch |
`2.30.9` -> `2.30.10` |
| [software.amazon.awssdk:aws-core](https://aws.amazon.com/sdkforjava) |
dependencies | misk/gradle/libs.versions.toml | gradle | patch |
`2.30.9` -> `2.30.10` |
| [software.amazon.awssdk:bom](https://aws.amazon.com/sdkforjava) |
dependencies | misk/gradle/libs.versions.toml | gradle | patch |
`2.30.9` -> `2.30.10` |
| [software.amazon.awssdk:auth](https://aws.amazon.com/sdkforjava) |
dependencies | misk/gradle/libs.versions.toml | gradle | patch |
`2.30.9` -> `2.30.10` |

---

### Release Notes

<details>
<summary>googleapis/java-datastore
(com.google.cloud:google-cloud-datastore)</summary>

###
[`v2.26.0`](https://github.com/googleapis/java-datastore/blob/HEAD/CHANGELOG.md#2260-2025-01-29)

##### Features

- Add firestoreInDatastoreMode for datastore emulator
([#&#8203;1698](googleapis/java-datastore#1698))
([50f106d](googleapis/java-datastore@50f106d))

##### Dependencies

- Update dependency com.google.cloud:sdk-platform-java-config to v3.42.0
([#&#8203;1725](googleapis/java-datastore#1725))
([1cbaf22](googleapis/java-datastore@1cbaf22))

</details>

<details>
<summary>datadog/dd-trace-java (com.datadoghq:dd-trace-api)</summary>

###
[`v1.46.0`](https://github.com/DataDog/dd-trace-java/releases/tag/v1.46.0):
1.46.0

##### Breaking Changes

> \[!WARNING]
> jnr-unixsocket is now an external dependency of dd-trace-ot and must
be included when deploying dd-trace-ot.

> \[!NOTE]
> The API `TracerScope.setAsync(boolean)`, used to manually control
asynchronous span propagation, does no more apply to the scope instance
but to the active span scope.

##### Components

##### Application Security Management (IAST)

- 🐛 Fix String.replace instrumentation for IAST
([#&#8203;8281](DataDog/dd-trace-java#8281) -
[@&#8203;Mariovido](https://github.com/Mariovido))
- ✨ Apply the standard nomenclature to the stacktrace configs
([#&#8203;8244](DataDog/dd-trace-java#8244) -
[@&#8203;jandro996](https://github.com/jandro996))
- 🐛 Exclude false positive weak randomness
([#&#8203;8232](DataDog/dd-trace-java#8232) -
[@&#8203;jandro996](https://github.com/jandro996))
- ✨ Propagation of translateEscapes of String class
([#&#8203;8186](DataDog/dd-trace-java#8186) -
[@&#8203;sezen-datadog](https://github.com/sezen-datadog))
- ✨ Add security control metrics
([#&#8203;8175](DataDog/dd-trace-java#8175) -
[@&#8203;jandro996](https://github.com/jandro996))
- ✨ Increase IAST propagation to StringBuffer setLength
([#&#8203;8128](DataDog/dd-trace-java#8128) -
[@&#8203;Mariovido](https://github.com/Mariovido))
- ✨ Add IAST taint tracking for DB values
([#&#8203;8072](DataDog/dd-trace-java#8072) -
[@&#8203;Mariovido](https://github.com/Mariovido))

##### Application Security Management (WAF)

- 🐛 Prevents a NPE when there is no subscriber for user events
([#&#8203;8258](DataDog/dd-trace-java#8258) -
[@&#8203;manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez))
- ✨ Apply the standard nomenclature to the stacktrace configs
([#&#8203;8244](DataDog/dd-trace-java#8244) -
[@&#8203;jandro996](https://github.com/jandro996))
- 🐛 Ensure cached subscriptions are cleared on reconfiguration via
RC ([#&#8203;8229](DataDog/dd-trace-java#8229)
-
[@&#8203;manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez))
- ✨ Add support for session tracking in Vertx
([#&#8203;8167](DataDog/dd-trace-java#8167) -
[@&#8203;manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez))
- ✨ Create span tag: \_dd.appsec.rasp.timeout
([#&#8203;8269](DataDog/dd-trace-java#8269) -
[@&#8203;Mariovido](https://github.com/Mariovido))

##### Build & Tooling

- 🐛 Ensure shaded helpers have unique names when injected into
class-loaders
([#&#8203;8192](DataDog/dd-trace-java#8192) -
[@&#8203;mcculls](https://github.com/mcculls))

##### Configuration at Runtime

- 🐛 Remove filtering of `DD_SERVICE` and `DD_ENV` from the tracer
([#&#8203;8176](DataDog/dd-trace-java#8176) -
[@&#8203;mhlidd](https://github.com/mhlidd))

##### Continuous Integration Visibility

- 🧹 Generalize TestRetryPolicy to TestExecutionPolicy
([#&#8203;8302](DataDog/dd-trace-java#8302) -
[@&#8203;nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog))
- 🧹 Parallelize CI Visibility settings requests
([#&#8203;8299](DataDog/dd-trace-java#8299) -
[@&#8203;nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog))
- 🧹 Generalize test retry logic
([#&#8203;8289](DataDog/dd-trace-java#8289) -
[@&#8203;nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog))
- 🧹 Generalize tests skipping logic
([#&#8203;8288](DataDog/dd-trace-java#8288) -
[@&#8203;nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog))
- 🧹 Remove skip and shouldBeSkipped methods from TestEventsHandler
in favor of isSkippable
([#&#8203;8286](DataDog/dd-trace-java#8286) -
[@&#8203;nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog))
- ✨⚡ Optimize Git repository information computation
([#&#8203;8270](DataDog/dd-trace-java#8270) -
[@&#8203;dougqh](https://github.com/dougqh))
- ✨ Always request known tests from the backend
([#&#8203;8268](DataDog/dd-trace-java#8268) -
[@&#8203;nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog))
- ✨ Fix NPE when trying to get retry analyzer in Test NG
([#&#8203;8253](DataDog/dd-trace-java#8253) -
[@&#8203;nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog))
- 🐛 Set test framework and test framework version tags atomically
([#&#8203;8252](DataDog/dd-trace-java#8252) -
[@&#8203;nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog))
- ✨ Add debug logging to Android Gradle module layout logic
([#&#8203;8251](DataDog/dd-trace-java#8251) -
[@&#8203;nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog))
- 🐛 Fix source and destination folders computation for Android
Gradle projects
([#&#8203;8190](DataDog/dd-trace-java#8190) -
[@&#8203;nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog))
- ✨ Add basic Scala Weaver sbt support
([#&#8203;8189](DataDog/dd-trace-java#8189) -
[@&#8203;daniel-mohedano](https://github.com/daniel-mohedano))
- ✨ Implement impacted tests detection
([#&#8203;8188](DataDog/dd-trace-java#8188) -
[@&#8203;nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog))

##### Data Streams Monitoring

- ✨ Change hash computation for protobuf to better represent
impacting changes + save proto number in schema
([#&#8203;8201](DataDog/dd-trace-java#8201) -
[@&#8203;vandonr](https://github.com/vandonr))

##### Database Monitoring

- Add peer service tag in dbm sql commenter
([#&#8203;7913](DataDog/dd-trace-java#7913) -
[@&#8203;jordan-wong](https://github.com/jordan-wong))

##### Dynamic Instrumentation

- ✨ Add support for SymDB to scan directories
([#&#8203;8306](DataDog/dd-trace-java#8306) -
[@&#8203;jpbempel](https://github.com/jpbempel))
- ✨ Add SymDB report for any jar scanning failures
([#&#8203;8300](DataDog/dd-trace-java#8300) -
[@&#8203;jpbempel](https://github.com/jpbempel))
- ✨ Use two budgets depending on type
([#&#8203;8283](DataDog/dd-trace-java#8283) -
[@&#8203;evanchooly](https://github.com/evanchooly))
- ✨ Institute a 10 snapshot per probe per trace budget
([#&#8203;8277](DataDog/dd-trace-java#8277) -
[@&#8203;evanchooly](https://github.com/evanchooly))
- 🐛 Avoid double snapshots for Exception Replay
([#&#8203;8273](DataDog/dd-trace-java#8273) -
[@&#8203;jpbempel](https://github.com/jpbempel))
- ✨ Simplify code origins. Separate out snapshot generation.
([#&#8203;8263](DataDog/dd-trace-java#8263) -
[@&#8203;evanchooly](https://github.com/evanchooly))
- ✨ Add Exception probe custom instrumentation
([#&#8203;8230](DataDog/dd-trace-java#8230) -
[@&#8203;jpbempel](https://github.com/jpbempel))
- ✨ Enhance log probes to honor debug session tags
([#&#8203;8215](DataDog/dd-trace-java#8215) -
[@&#8203;evanchooly](https://github.com/evanchooly))
- 🐛 Don't redact env tokens from debugger probe snapshots
([#&#8203;8211](DataDog/dd-trace-java#8211) -
[@&#8203;watson](https://github.com/watson))
- ✨⚡ Move Trace/SpanId capture at commit time
([#&#8203;8184](DataDog/dd-trace-java#8184) -
[@&#8203;jpbempel](https://github.com/jpbempel))
- 🐛 Capture values at entry for method probe
([#&#8203;8169](DataDog/dd-trace-java#8169) -
[@&#8203;jpbempel](https://github.com/jpbempel))

##### JMX fetch

- 🐛 Mute JMXFetch Shutdown in progress error
([#&#8203;8068](DataDog/dd-trace-java#8068) -
[@&#8203;ygree](https://github.com/ygree))

##### OpenTracing

- ⚠️🧹 Make jnr-unixsocket an explicit dependency of
dd-trace-ot
([#&#8203;8307](DataDog/dd-trace-java#8307) -
[@&#8203;mcculls](https://github.com/mcculls))

##### Profiling

- 🐛 Avoid unsupported API call for creating folders on windows
([#&#8203;8304](DataDog/dd-trace-java#8304) -
[@&#8203;jbachorik](https://github.com/jbachorik))
- ✨ Tag profiles for serverless
([#&#8203;8279](DataDog/dd-trace-java#8279) -
[@&#8203;jbachorik](https://github.com/jbachorik))
- ✨ add queue type and length to queue events
([#&#8203;8242](DataDog/dd-trace-java#8242) -
[@&#8203;richardstartin](https://github.com/richardstartin))
- 🐛 TempLocationManager Fixes and Improvements
([#&#8203;8191](DataDog/dd-trace-java#8191) -
[@&#8203;jbachorik](https://github.com/jbachorik))
- ✨ Bump ddprof to 1.18.0
([#&#8203;8173](DataDog/dd-trace-java#8173) -
[@&#8203;jbachorik](https://github.com/jbachorik))
- ✨ Report profiler initialization and configuration errors to
telemetry
([#&#8203;8171](DataDog/dd-trace-java#8171) -
[@&#8203;jbachorik](https://github.com/jbachorik))

##### Telemetry

- ✨ Add pending traces report in tracer flares
([#&#8203;8053](DataDog/dd-trace-java#8053) -
[@&#8203;mhlidd](https://github.com/mhlidd))

##### Testing

- ✨ Test http server requests in parallel
([#&#8203;8222](DataDog/dd-trace-java#8222) -
[@&#8203;amarziali](https://github.com/amarziali))

##### Trace context propagation

- ✨ Add non default propagator registration
([#&#8203;8310](DataDog/dd-trace-java#8310) -
[@&#8203;PerfectSlayer](https://github.com/PerfectSlayer))

##### Tracer core

- ✨ Probe for existence of IBMSASL or ACCP security providers
([#&#8203;8276](DataDog/dd-trace-java#8276) -
[@&#8203;mcculls](https://github.com/mcculls))
- ✨⚡ Overhead improvement to agent feedback based sampling
([#&#8203;8265](DataDog/dd-trace-java#8265) -
[@&#8203;dougqh](https://github.com/dougqh))
- 🧹 Move async propagation API from scope to tracer
([#&#8203;8231](DataDog/dd-trace-java#8231) -
[@&#8203;PerfectSlayer](https://github.com/PerfectSlayer))
- ✨ Introduce context propagation API
([#&#8203;8161](DataDog/dd-trace-java#8161) -
[@&#8203;PerfectSlayer](https://github.com/PerfectSlayer))
- ✨🧪 Use env-entry to add tags per webapp deployment
([#&#8203;8138](DataDog/dd-trace-java#8138) -
[@&#8203;amarziali](https://github.com/amarziali))
- ✨ Introduce context helpers API
([#&#8203;8134](DataDog/dd-trace-java#8134) -
[@&#8203;PerfectSlayer](https://github.com/PerfectSlayer))
- ✨ Support IPv6 values for `DD_AGENT_HOST` and
`DD_TRACE_AGENT_URL`
([#&#8203;7984](DataDog/dd-trace-java#7984) -
[@&#8203;mhlidd](https://github.com/mhlidd))

##### Instrumentations

##### Apache HttpComponents

- 🐛 Properly finish spans and support latest apache httpclient5
([#&#8203;8272](DataDog/dd-trace-java#8272) -
[@&#8203;amarziali](https://github.com/amarziali))

##### AWS Lambda instrumentation

- 🐛 Properly capture lambda payloads for all handler types.
([#&#8203;8264](DataDog/dd-trace-java#8264) -
[@&#8203;purple4reina](https://github.com/purple4reina))

##### AWS S3 instrumentation

- 💡 Create S3 instrumentation + add span pointers
([#&#8203;8075](DataDog/dd-trace-java#8075) -
[@&#8203;nhulston](https://github.com/nhulston))

##### AWS SDK instrumentation

- 🐛 Revert "Add avoid double instrumenting lambda non-streaming
handlers."
([#&#8203;8247](DataDog/dd-trace-java#8247) -
[@&#8203;nhulston](https://github.com/nhulston))

##### Cassandra

- ✨ Allow extracting keyspace from statement result
([#&#8203;8239](DataDog/dd-trace-java#8239) -
[@&#8203;amarziali](https://github.com/amarziali))

##### Core Java language instrumentation

- ✨ Propagation of translateEscapes of String class
([#&#8203;8186](DataDog/dd-trace-java#8186) -
[@&#8203;sezen-datadog](https://github.com/sezen-datadog))

##### Eclipse Vert.x instrumentation

- 🐛 Fix vertx worker propagation and error handling
([#&#8203;8237](DataDog/dd-trace-java#8237) -
[@&#8203;amarziali](https://github.com/amarziali))
- ✨ Support vertx 5
([#&#8203;8220](DataDog/dd-trace-java#8220) -
[@&#8203;amarziali](https://github.com/amarziali))
- ✨ Add support for session tracking in Vertx
([#&#8203;8167](DataDog/dd-trace-java#8167) -
[@&#8203;manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez))

##### Kafka instrumentation

- 🐛 Prevent possible NPE calculating Kafka record header size
([#&#8203;8292](DataDog/dd-trace-java#8292) -
[@&#8203;ygree](https://github.com/ygree))

##### Mule instrumentation

- 🐛 Fix crash using Mule with JPMS
([#&#8203;8187](DataDog/dd-trace-java#8187) -
[@&#8203;amarziali](https://github.com/amarziali))

##### Protocol Buffer instrumentation

- ✨ Change hash computation for protobuf to better represent
impacting changes + save proto number in schema
([#&#8203;8201](DataDog/dd-trace-java#8201) -
[@&#8203;vandonr](https://github.com/vandonr))

##### Spring instrumentation

- 🐛 Preserve getQualifier from spring scheduling runnables
([#&#8203;8293](DataDog/dd-trace-java#8293) -
[@&#8203;amarziali](https://github.com/amarziali))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am
every weekday" in timezone Australia/Melbourne, Automerge - At any time
(no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config help](https://github.com/renovatebot/renovate/discussions) if
that's undesired.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Renovate
Bot](https://github.com/renovatebot/renovate).

GitOrigin-RevId: bb09d47e4eed77a003f630273b4d0a84003eb899
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp: ci visibility Continuous Integration Visibility type: refactoring
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants