[SPARK-53164][CORE][K8S][DSTREAM] Use Java Files.readAllBytes instead of Files.toByteArray
#51891
Conversation
…ad of `Files.toByteArray`
| assert(diskStore.getSize(blockId) === testData.length) | ||
|
|
||
| val diskData = Files.toByteArray(diskBlockManager.getFile(blockId.name)) | ||
| val diskData = Files.readAllBytes(diskBlockManager.getFile(blockId.name).toPath) |
There was a problem hiding this comment.
Which one is better, Files.readAllBytes or com.google.common.io.ByteStreams#toByteArray(java.io.InputStream)?
There was a problem hiding this comment.
I checked the performance. It's the same for reading part, @LuciferYang . The main performance difference happens at writer API part usually.
There was a problem hiding this comment.
So, I mentioned "The built-in Java method is as good as 3rd party library." in this PR description.
There was a problem hiding this comment.
BTW, FYI, I'm digging this are as a part of Java 25 preparation and CI runtime improvement. If we stick to the old libraries, we cannot get the benefit of new Java version's improvement. For me, the stale Scala 2.13 and 3rd party libraries are very suspicious to me.
|
BTW, all tests passed except one flaky test of |
|
Thank you, @LuciferYang . Merged to master for Apache Spark 4.1.0. |
2 participants
What changes were proposed in this pull request?
This PR aims to use Java 9+
java.nio.file.Files.readAllBytesinstead ofcom.google.common.io.Files.toByteArray.In addition, a new Scalastyle rule is added to ban
Files.toByteArrayfor consistency.Why are the changes needed?
The built-in Java method is as good as 3rd party library.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass the CIs.
Was this patch authored or co-authored using generative AI tooling?
No.