Skip to content

Conversation

smola
Copy link
Member

@smola smola commented Mar 5, 2025

What Does This Do

Change DD_API_SECURITY_ENABLED=true by default. This should have impact mostly only when AppSec is enabled, except for the fact a few more classes are loaded earlier during AppSec initialization (that accounts for the small startup time regression. And a small impact on trace serialization since this adds a callback on every serialized span (which is short-circuited when AppSec is disabled).

This feature es effectively enabled only if AppSec is also enabled.

Motivation

API Security is now core functionality to the App & API Protection (AppSec), so we want it to be available by default to all AppSec customers.

Additional Notes

Contributor Checklist

Jira ticket: APPSEC-57850

@smola smola added comp: asm waf Application Security Management (WAF) type: enhancement Enhancements and improvements labels Mar 5, 2025
@pr-commenter
Copy link

pr-commenter bot commented Mar 5, 2025

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/enable-api-security-by-default
git_commit_date 1750053828 1750054797
git_commit_sha b1b0ab3 0663d5813d
release_version 1.50.0-SNAPSHOT~b1b0ab330e 1.50.0-SNAPSHOT~c0663d5813d
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1750056423 1750056423
ci_job_id 982775542 982775542
ci_pipeline_id 67803366 67803366
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-symkarcw-project-304-concurrent-0-d0d59wjh 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-symkarcw-project-304-concurrent-0-d0d59wjh 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 1 performance regressions! Performance is the same for 58 metrics, 12 unstable metrics.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:startup:petclinic:tracing:AppSec worse
[+2.786ms; +7.562ms] or [+5.014%; +13.607%]
60.745ms 55.571ms
Startup time reports for petclinic
gantt
    title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~c0663d5813d, baseline=1.50.0-SNAPSHOT~b1b0ab330e

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.022 s) : 0, 1022423
Total [baseline] (10.493 s) : 0, 10493214
Agent [candidate] (1.027 s) : 0, 1026905
Total [candidate] (10.439 s) : 0, 10438708
section appsec
Agent [baseline] (1.182 s) : 0, 1182125
Total [baseline] (10.682 s) : 0, 10681994
Agent [candidate] (1.185 s) : 0, 1184547
Total [candidate] (10.703 s) : 0, 10702940
section iast
Agent [baseline] (1.16 s) : 0, 1160137
Total [baseline] (10.833 s) : 0, 10832913
Agent [candidate] (1.155 s) : 0, 1155256
Total [candidate] (10.83 s) : 0, 10829622
section profiling
Agent [baseline] (1.277 s) : 0, 1277132
Total [baseline] (10.908 s) : 0, 10908330
Agent [candidate] (1.27 s) : 0, 1269703
Total [candidate] (10.826 s) : 0, 10826267
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.022 s -
Agent appsec 1.182 s 159.702 ms (15.6%)
Agent iast 1.16 s 137.714 ms (13.5%)
Agent profiling 1.277 s 254.709 ms (24.9%)
Total tracing 10.493 s -
Total appsec 10.682 s 188.781 ms (1.8%)
Total iast 10.833 s 339.7 ms (3.2%)
Total profiling 10.908 s 415.117 ms (4.0%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.027 s -
Agent appsec 1.185 s 157.642 ms (15.4%)
Agent iast 1.155 s 128.351 ms (12.5%)
Agent profiling 1.27 s 242.797 ms (23.6%)
Total tracing 10.439 s -
Total appsec 10.703 s 264.232 ms (2.5%)
Total iast 10.83 s 390.914 ms (3.7%)
Total profiling 10.826 s 387.559 ms (3.7%)
gantt
    title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~c0663d5813d, baseline=1.50.0-SNAPSHOT~b1b0ab330e

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (682.503 ms) : 0, 682503
BytebuddyAgent [candidate] (682.826 ms) : 0, 682826
GlobalTracer [baseline] (240.172 ms) : 0, 240172
GlobalTracer [candidate] (240.655 ms) : 0, 240655
AppSec [baseline] (55.571 ms) : 0, 55571
AppSec [candidate] (60.745 ms) : 0, 60745
Debugger [baseline] (6.175 ms) : 0, 6175
Debugger [candidate] (6.154 ms) : 0, 6154
Remote Config [baseline] (735.153 µs) : 0, 735
Remote Config [candidate] (763.648 µs) : 0, 764
Telemetry [baseline] (13.601 ms) : 0, 13601
Telemetry [candidate] (12.116 ms) : 0, 12116
section appsec
BytebuddyAgent [baseline] (709.945 ms) : 0, 709945
BytebuddyAgent [candidate] (709.752 ms) : 0, 709752
GlobalTracer [baseline] (236.254 ms) : 0, 236254
GlobalTracer [candidate] (235.518 ms) : 0, 235518
IAST [baseline] (21.996 ms) : 0, 21996
IAST [candidate] (21.922 ms) : 0, 21922
AppSec [baseline] (176.335 ms) : 0, 176335
AppSec [candidate] (179.987 ms) : 0, 179987
Debugger [baseline] (5.925 ms) : 0, 5925
Debugger [candidate] (5.845 ms) : 0, 5845
Remote Config [baseline] (622.358 µs) : 0, 622
Remote Config [candidate] (618.384 µs) : 0, 618
Telemetry [baseline] (7.338 ms) : 0, 7338
Telemetry [candidate] (7.265 ms) : 0, 7265
section iast
BytebuddyAgent [baseline] (809.163 ms) : 0, 809163
BytebuddyAgent [candidate] (803.481 ms) : 0, 803481
GlobalTracer [baseline] (232.635 ms) : 0, 232635
GlobalTracer [candidate] (230.935 ms) : 0, 230935
IAST [baseline] (29.336 ms) : 0, 29336
IAST [candidate] (27.035 ms) : 0, 27035
AppSec [baseline] (50.572 ms) : 0, 50572
AppSec [candidate] (55.601 ms) : 0, 55601
Debugger [baseline] (6.009 ms) : 0, 6009
Debugger [candidate] (6.043 ms) : 0, 6043
Remote Config [baseline] (601.374 µs) : 0, 601
Remote Config [candidate] (597.07 µs) : 0, 597
Telemetry [baseline] (8.07 ms) : 0, 8070
Telemetry [candidate] (7.904 ms) : 0, 7904
section profiling
BytebuddyAgent [baseline] (679.795 ms) : 0, 679795
BytebuddyAgent [candidate] (674.352 ms) : 0, 674352
GlobalTracer [baseline] (363.781 ms) : 0, 363781
GlobalTracer [candidate] (360.021 ms) : 0, 360021
AppSec [baseline] (61.955 ms) : 0, 61955
AppSec [candidate] (63.739 ms) : 0, 63739
Debugger [baseline] (6.16 ms) : 0, 6160
Debugger [candidate] (6.079 ms) : 0, 6079
Remote Config [baseline] (662.133 µs) : 0, 662
Remote Config [candidate] (655.104 µs) : 0, 655
Telemetry [baseline] (8.3 ms) : 0, 8300
Telemetry [candidate] (8.105 ms) : 0, 8105
ProfilingAgent [baseline] (105.122 ms) : 0, 105122
ProfilingAgent [candidate] (105.929 ms) : 0, 105929
Profiling [baseline] (105.147 ms) : 0, 105147
Profiling [candidate] (105.954 ms) : 0, 105954
Loading
Startup time reports for insecure-bank
gantt
    title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~c0663d5813d, baseline=1.50.0-SNAPSHOT~b1b0ab330e

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.039 s) : 0, 1039141
Total [baseline] (8.648 s) : 0, 8647901
Agent [candidate] (1.028 s) : 0, 1028180
Total [candidate] (8.572 s) : 0, 8571902
section iast
Agent [baseline] (1.148 s) : 0, 1148423
Total [baseline] (9.234 s) : 0, 9234249
Agent [candidate] (1.151 s) : 0, 1150770
Total [candidate] (9.212 s) : 0, 9212249
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.15 s) : 0, 1149578
Total [baseline] (9.241 s) : 0, 9241318
Agent [candidate] (1.154 s) : 0, 1154243
Total [candidate] (9.219 s) : 0, 9219371
section iast_TELEMETRY_OFF
Agent [baseline] (1.148 s) : 0, 1147973
Total [baseline] (9.287 s) : 0, 9286780
Agent [candidate] (1.149 s) : 0, 1148919
Total [candidate] (9.195 s) : 0, 9195301
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.039 s -
Agent iast 1.148 s 109.282 ms (10.5%)
Agent iast_HARDCODED_SECRET_DISABLED 1.15 s 110.436 ms (10.6%)
Agent iast_TELEMETRY_OFF 1.148 s 108.832 ms (10.5%)
Total tracing 8.648 s -
Total iast 9.234 s 586.348 ms (6.8%)
Total iast_HARDCODED_SECRET_DISABLED 9.241 s 593.417 ms (6.9%)
Total iast_TELEMETRY_OFF 9.287 s 638.879 ms (7.4%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.028 s -
Agent iast 1.151 s 122.59 ms (11.9%)
Agent iast_HARDCODED_SECRET_DISABLED 1.154 s 126.063 ms (12.3%)
Agent iast_TELEMETRY_OFF 1.149 s 120.738 ms (11.7%)
Total tracing 8.572 s -
Total iast 9.212 s 640.347 ms (7.5%)
Total iast_HARDCODED_SECRET_DISABLED 9.219 s 647.469 ms (7.6%)
Total iast_TELEMETRY_OFF 9.195 s 623.399 ms (7.3%)
gantt
    title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~c0663d5813d, baseline=1.50.0-SNAPSHOT~b1b0ab330e

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (694.304 ms) : 0, 694304
BytebuddyAgent [candidate] (685.16 ms) : 0, 685160
GlobalTracer [baseline] (242.873 ms) : 0, 242873
GlobalTracer [candidate] (240.388 ms) : 0, 240388
AppSec [baseline] (56.424 ms) : 0, 56424
AppSec [candidate] (59.183 ms) : 0, 59183
Debugger [baseline] (6.346 ms) : 0, 6346
Debugger [candidate] (6.212 ms) : 0, 6212
Remote Config [baseline] (754.745 µs) : 0, 755
Remote Config [candidate] (744.078 µs) : 0, 744
Telemetry [baseline] (14.581 ms) : 0, 14581
Telemetry [candidate] (12.807 ms) : 0, 12807
section iast
BytebuddyAgent [baseline] (800.951 ms) : 0, 800951
BytebuddyAgent [candidate] (800.558 ms) : 0, 800558
GlobalTracer [baseline] (229.848 ms) : 0, 229848
GlobalTracer [candidate] (230.187 ms) : 0, 230187
IAST [baseline] (26.75 ms) : 0, 26750
IAST [candidate] (25.891 ms) : 0, 25891
AppSec [baseline] (52.802 ms) : 0, 52802
AppSec [candidate] (56.32 ms) : 0, 56320
Debugger [baseline] (6.01 ms) : 0, 6010
Debugger [candidate] (5.891 ms) : 0, 5891
Remote Config [baseline] (606.364 µs) : 0, 606
Remote Config [candidate] (590.676 µs) : 0, 591
Telemetry [baseline] (7.977 ms) : 0, 7977
Telemetry [candidate] (7.832 ms) : 0, 7832
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (800.497 ms) : 0, 800497
BytebuddyAgent [candidate] (802.293 ms) : 0, 802293
GlobalTracer [baseline] (230.579 ms) : 0, 230579
GlobalTracer [candidate] (231.127 ms) : 0, 231127
IAST [baseline] (25.488 ms) : 0, 25488
IAST [candidate] (25.325 ms) : 0, 25325
AppSec [baseline] (54.846 ms) : 0, 54846
AppSec [candidate] (57.327 ms) : 0, 57327
Debugger [baseline] (6.04 ms) : 0, 6040
Debugger [candidate] (6.041 ms) : 0, 6041
Remote Config [baseline] (597.376 µs) : 0, 597
Remote Config [candidate] (609.84 µs) : 0, 610
Telemetry [baseline] (8.049 ms) : 0, 8049
Telemetry [candidate] (7.973 ms) : 0, 7973
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (799.747 ms) : 0, 799747
BytebuddyAgent [candidate] (798.408 ms) : 0, 798408
GlobalTracer [baseline] (230.787 ms) : 0, 230787
GlobalTracer [candidate] (230.984 ms) : 0, 230984
IAST [baseline] (27.263 ms) : 0, 27263
IAST [candidate] (28.652 ms) : 0, 28652
AppSec [baseline] (52.185 ms) : 0, 52185
AppSec [candidate] (52.17 ms) : 0, 52170
Debugger [baseline] (5.996 ms) : 0, 5996
Debugger [candidate] (5.97 ms) : 0, 5970
Remote Config [baseline] (609.318 µs) : 0, 609
Remote Config [candidate] (596.778 µs) : 0, 597
Telemetry [baseline] (7.848 ms) : 0, 7848
Telemetry [candidate] (7.913 ms) : 0, 7913
Loading

Load

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/enable-api-security-by-default
git_commit_date 1750053828 1750054803
git_commit_sha b1b0ab3 0663d5813d
release_version 1.50.0-SNAPSHOT~b1b0ab330e 1.50.0-SNAPSHOT~c0663d5813d
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1750056212 1750056212
ci_job_id 982775544 982775544
ci_pipeline_id 67803366 67803366
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-pcisq8qe-project-304-concurrent-1-iuoswfsm 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-pcisq8qe-project-304-concurrent-1-iuoswfsm 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava
gantt
    title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~c0663d5813d, baseline=1.50.0-SNAPSHOT~b1b0ab330e
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.757 s) : 15757000, 15757000
.   : milestone, 15757000,
appsec (14.77 s) : 14770000, 14770000
.   : milestone, 14770000,
iast (18.978 s) : 18978000, 18978000
.   : milestone, 18978000,
iast_GLOBAL (18.132 s) : 18132000, 18132000
.   : milestone, 18132000,
profiling (15.373 s) : 15373000, 15373000
.   : milestone, 15373000,
tracing (14.971 s) : 14971000, 14971000
.   : milestone, 14971000,
section candidate
no_agent (15.288 s) : 15288000, 15288000
.   : milestone, 15288000,
appsec (14.946 s) : 14946000, 14946000
.   : milestone, 14946000,
iast (18.653 s) : 18653000, 18653000
.   : milestone, 18653000,
iast_GLOBAL (18.138 s) : 18138000, 18138000
.   : milestone, 18138000,
profiling (15.326 s) : 15326000, 15326000
.   : milestone, 15326000,
tracing (14.936 s) : 14936000, 14936000
.   : milestone, 14936000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.757 s [15.757 s, 15.757 s] -
appsec 14.77 s [14.77 s, 14.77 s] -987.0 ms (-6.3%)
iast 18.978 s [18.978 s, 18.978 s] 3.221 s (20.4%)
iast_GLOBAL 18.132 s [18.132 s, 18.132 s] 2.375 s (15.1%)
profiling 15.373 s [15.373 s, 15.373 s] -384.0 ms (-2.4%)
tracing 14.971 s [14.971 s, 14.971 s] -786.0 ms (-5.0%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.288 s [15.288 s, 15.288 s] -
appsec 14.946 s [14.946 s, 14.946 s] -342.0 ms (-2.2%)
iast 18.653 s [18.653 s, 18.653 s] 3.365 s (22.0%)
iast_GLOBAL 18.138 s [18.138 s, 18.138 s] 2.85 s (18.6%)
profiling 15.326 s [15.326 s, 15.326 s] 38.0 ms (0.2%)
tracing 14.936 s [14.936 s, 14.936 s] -352.0 ms (-2.3%)
Execution time for tomcat
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~c0663d5813d, baseline=1.50.0-SNAPSHOT~b1b0ab330e
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.481 ms) : 1469, 1492
.   : milestone, 1481,
appsec (2.419 ms) : 2370, 2469
.   : milestone, 2419,
iast (2.198 ms) : 2136, 2259
.   : milestone, 2198,
iast_GLOBAL (2.237 ms) : 2175, 2298
.   : milestone, 2237,
profiling (2.036 ms) : 1987, 2085
.   : milestone, 2036,
tracing (2.008 ms) : 1960, 2055
.   : milestone, 2008,
section candidate
no_agent (1.482 ms) : 1471, 1494
.   : milestone, 1482,
appsec (2.417 ms) : 2368, 2466
.   : milestone, 2417,
iast (2.194 ms) : 2132, 2256
.   : milestone, 2194,
iast_GLOBAL (2.237 ms) : 2176, 2299
.   : milestone, 2237,
profiling (2.039 ms) : 1989, 2088
.   : milestone, 2039,
tracing (2.01 ms) : 1962, 2058
.   : milestone, 2010,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.481 ms [1.469 ms, 1.492 ms] -
appsec 2.419 ms [2.37 ms, 2.469 ms] 938.766 µs (63.4%)
iast 2.198 ms [2.136 ms, 2.259 ms] 716.865 µs (48.4%)
iast_GLOBAL 2.237 ms [2.175 ms, 2.298 ms] 755.83 µs (51.0%)
profiling 2.036 ms [1.987 ms, 2.085 ms] 555.453 µs (37.5%)
tracing 2.008 ms [1.96 ms, 2.055 ms] 526.816 µs (35.6%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.482 ms [1.471 ms, 1.494 ms] -
appsec 2.417 ms [2.368 ms, 2.466 ms] 934.877 µs (63.1%)
iast 2.194 ms [2.132 ms, 2.256 ms] 711.732 µs (48.0%)
iast_GLOBAL 2.237 ms [2.176 ms, 2.299 ms] 755.119 µs (50.9%)
profiling 2.039 ms [1.989 ms, 2.088 ms] 556.537 µs (37.5%)
tracing 2.01 ms [1.962 ms, 2.058 ms] 527.945 µs (35.6%)

@smola smola force-pushed the smola/enable-api-security-by-default branch from 70381c5 to 82de3c8 Compare May 13, 2025 14:59
@smola smola marked this pull request as ready for review June 3, 2025 13:51
@smola smola requested a review from a team as a code owner June 3, 2025 13:51
@smola smola requested a review from mcculls June 3, 2025 13:51
@dougqh
Copy link
Contributor

dougqh commented Jun 3, 2025

Can you explain a bit more what API security does?
While I see that there no load regressions, I still like to better understand the implications on the performance of the default configuration.

@smola
Copy link
Member Author

smola commented Jun 5, 2025

@dougqh Long story short: it does schema inference of requests. This is done with sampling, and happens late in the request cycle (in a trace post-processor). The whole thing can be seen at #8178

For the default case, there's two things impacting here:

  • Some additional early loaded classes within the appsec module. I think the impact here is probably negligible.
  • At runtime, there's this new code block for the default case:

final SpanPostProcessor postProcessor = SpanPostProcessor.Holder.INSTANCE;
try {
final long timeout = Config.get().getTracePostProcessingTimeout();
final long deadline = System.currentTimeMillis() + timeout;
final boolean[] timedOut = {false};
final BooleanSupplier timeoutCheck =
() -> {
if (timedOut[0]) {
return true;
}
if (System.currentTimeMillis() > deadline) {
timedOut[0] = true;
}
return timedOut[0];
};
for (DDSpan span : trace) {
postProcessor.process(span, timeoutCheck);
}
} catch (Throwable e) {
log.debug("Error while trace post-processing", e);
}

Our trace post-processor is initialized by default (to avoid complicated things too much with runtime activation), and it would be short-circuited here:

final RequestContext ctx_ = span.getRequestContext();
if (ctx_ == null) {
return;
}
final AppSecRequestContext ctx = ctx_.getData(RequestContextSlot.APPSEC);
if (ctx == null) {
return;
}

@smola smola marked this pull request as draft June 11, 2025 09:05
@smola
Copy link
Member Author

smola commented Jun 11, 2025

Given the performance regressions, I moved this back to draft. I will open another PR with performance improvements.

@smola smola marked this pull request as ready for review June 16, 2025 06:06
@smola
Copy link
Member Author

smola commented Jun 16, 2025

Regressions in the load benchmark we're spurious, merging.

@smola smola merged commit 2885767 into master Jun 16, 2025
485 of 486 checks passed
@smola smola deleted the smola/enable-api-security-by-default branch June 16, 2025 07:25
@github-actions github-actions bot added this to the 1.50.0 milestone Jun 16, 2025
sarahchen6 pushed a commit that referenced this pull request Jun 18, 2025
sarahchen6 added a commit that referenced this pull request Jun 18, 2025
sarahchen6 added a commit that referenced this pull request Jun 19, 2025
sarahchen6 added a commit that referenced this pull request Jun 19, 2025
smola added a commit that referenced this pull request Jun 19, 2025
smola added a commit that referenced this pull request Jun 19, 2025
smola added a commit that referenced this pull request Jun 19, 2025
@PerfectSlayer PerfectSlayer added the tag: no release notes Changes to exclude from release notes label Jun 20, 2025
@PerfectSlayer
Copy link
Contributor

Removing it from changelog as it was reverted for 1.50.0

smola added a commit that referenced this pull request Jun 20, 2025
* Lazy API Security initialization
* Reapply "Enable API Security by default (#8511)" (#9006)

This reverts commit 2f4c864.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: asm waf Application Security Management (WAF) tag: no release notes Changes to exclude from release notes type: enhancement Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants