8365588: defineClass that accepts a ByteBuffer does not work as expected

[xuemingshen-oracle]

Background

• ClassLoader.defineClass can receive class data in the form of arrays or ByteBuffers.
• For array-backed data (defineClass1), a defensive copy is made before passing it to JVM_DefineClassWithSource().
• For Direct-ByteBuffer variants (defineClass2), no defensive copy is made, which creates a risk that the underlying bytes could be modified while the JVM is processing them.
• Although a caller could always modify a buffer before a defensive copy is made — a race condition that cannot be completely prevented — the main concern is ensuring that the JVM never processes class bytes that are being concurrently modified.

Problem

• Concurrent modification risk during processing: while we cannot prevent pre-copy modifications, we must prevent the JVM from using class bytes that are being modified concurrently.
• Performance concerns: defensive copies have a cost, especially for direct byte buffers. Making copies unnecessarily for trusted class loaders (like the built-in class loader) would hurt performance.

Solution

• Make a defensive copy of the direct byte-buffer only when the class loader is NOT a built-in/trusted class loader.
• For the built-in class loader, skip the copy because the JVM can guarantee that the buffer contents remain intact.

This approach ensures the integrity of class bytes processes for untrusted or custom class loaders while minimizing performance impact for trusted or built-in loaders.

Benchmark

A JMH benchmark has been added to measure the potential cost of the defensive copy. The results indicate that the performance impact is minimal and largely insignificant.

Before:

Benchmark                                               Mode  Cnt     Score      Error  Units
ClassLoaderDefineClass.testDefineClassByteBufferDirect  avgt   15  8387.247 ± 1405.681  ns/op
ClassLoaderDefineClass.testDefineClassByteBufferHeap    avgt   15  8971.739 ± 1020.288  ns/op
Finished running test 'micro:org.openjdk.bench.java.lang.ClassLoaderDefineClass'
Test report is stored in /Users/xuemingshen/jdk26/build/macosx-aarch64/test-results/micro_org_openjdk_bench_java_lang_ClassLoaderDefineClass

After:

Benchmark                                               Mode  Cnt     Score      Error  Units
ClassLoaderDefineClass.testDefineClassByteBufferDirect  avgt   15  8521.881 ± 2002.011  ns/op
ClassLoaderDefineClass.testDefineClassByteBufferHeap    avgt   15  9002.842 ± 1099.635  ns/op
Finished running test 'micro:org.openjdk.bench.java.lang.ClassLoaderDefineClass'
Test report is stored in /Users/xuemingshen/jdk26/build/macosx-aarch64/test-results/micro_org_openjdk_bench_java_lang_ClassLoaderDefineClass

Startup

./brr -b Startup-SwingSet,Startup3-Framer,Startup3-JettyStartup3-Noop -i 5 -j /Users/xuemingshen/jdk26
./brr compare xueming.shen.Base xueming.shen.JDK-8365588

Building brunte ($BRUNTE_REBUILD=)
Baseline: /Users/xuemingshen/Workspaces/brunte/opt/brunte/sandboxes/xueming.shen.Base/Startup3-Noop
    Test: /Users/xuemingshen/Workspaces/brunte/opt/brunte/sandboxes/xueming.shen.JDK-8365588/Startup3-Noop

Name          Cnt  Base   Error   Test   Error  Unit  Change
Startup3-Noop   5 0.030 ± 0.000  0.030 ± 0.000 lbOps   1.00x (p = 0.000 )
  :Noop           0.030 ± 0.000  0.030 ± 0.000 lbOps   1.00x (p = 0.000 )
  :rw.runtime     1.600 ± 2.109  1.400 ± 2.109 lbOps   1.14x (p = 0.580 )
  * = significant

Baseline: /Users/xuemingshen/Workspaces/brunte/opt/brunte/sandboxes/xueming.shen.Base/Startup3-Framer
    Test: /Users/xuemingshen/Workspaces/brunte/opt/brunte/sandboxes/xueming.shen.JDK-8365588/Startup3-Framer

Name            Cnt  Base   Error   Test   Error  Unit  Change
Startup3-Framer   5 0.388 ± 0.634  0.260 ± 0.077 lbOps   1.49x (p = 0.157 )
  :Framer           0.388 ± 0.634  0.260 ± 0.077 lbOps   1.49x (p = 0.157 )
  :rw.runtime       1.600 ± 2.109  1.800 ± 1.722 lbOps   0.89x (p = 0.545 )
  * = significant

Baseline: /Users/xuemingshen/Workspaces/brunte/opt/brunte/sandboxes/xueming.shen.Base/Startup3-Jetty
    Test: /Users/xuemingshen/Workspaces/brunte/opt/brunte/sandboxes/xueming.shen.JDK-8365588/Startup3-Jetty

Name           Cnt  Base   Error   Test   Error  Unit  Change
Startup3-Jetty   5 0.132 ± 0.017  0.134 ± 0.021 lbOps   0.99x (p = 0.545 )
  :jetty           0.132 ± 0.017  0.134 ± 0.021 lbOps   0.99x (p = 0.545 )
  :rw.runtime      1.600 ± 2.109  1.800 ± 1.722 lbOps   0.89x (p = 0.545 )
  * = significant

Baseline: /Users/xuemingshen/Workspaces/brunte/opt/brunte/sandboxes/xueming.shen.Base/Startup-SwingSet	 -b Startup-SwingSet -j /Users/xuemingshen/jdk26/build/macosx-aarch64/images/jdk -w microbenchmarks -i 5 --harness-bundle https://artifactory-cache-phx.java.oraclecorp.com/artifactory/jdk-virtual/com/oracle/swingset-startup/1.27/swingset-startup-1.27.jar
    Test: /Users/xuemingshen/Workspaces/brunte/opt/brunte/sandboxes/xueming.shen.JDK-8365588/Startup-SwingSet	 -b Startup-SwingSet -j /Users/xuemingshen/jdk26/build/macosx-aarch64/images/jdk -w microbenchmarks -i 5 --harness-bundle https://artifactory-cache-phx.java.oraclecorp.com/artifactory/jdk-virtual/com/oracle/swingset-startup/1.27/swingset-startup-1.27.jar

Name              Cnt    Base     Error     Test    Error  Unit  Change
measureSingleShot 100 287.620 ± 101.103  295.490 ± 98.585 ms/op   0.97x (p = 0.850 )
  * = significant

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8365588: defineClass that accepts a ByteBuffer does not work as expected (Bug - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/27569/head:pull/27569
$ git checkout pull/27569

Update a local copy of the PR:
$ git checkout pull/27569
$ git pull https://git.openjdk.org/jdk.git pull/27569/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 27569

View PR using the GUI difftool:
$ git pr show -t 27569

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/27569.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back sherman! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@xuemingshen-oracle The following label will be automatically applied to this pull request:

• core-libs

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 01: Full - Incremental (c7cb474f)
• 00: Full (cb294829)

[liach]

Do we need to acquire/release session on the base buffer too?

[openjdk]

@xuemingshen-oracle Please do not rebase or force-push to an active PR as it invalidates existing review comments. Note for future reference, the bots always squash all changes into a single commit automatically as part of the integration. See OpenJDK Developers’ Guide for more information.