[SPARK-24107][CORE][followup] ChunkedByteBuffer.writeFully method has not reset the limit value #21327
Conversation
|
Test build #90622 has finished for PR 21327 at commit
|
|
retest this please |
|
Test build #90628 has finished for PR 21327 at commit
|
| } finally { | ||
| bytes.limit(curChunkLimit) | ||
| } | ||
| // If `bytes` is an on-heap ByteBuffer, the JDK will copy it to a temporary direct |
There was a problem hiding this comment.
how about the JDK -> the JVM?
There was a problem hiding this comment.
the caching happens in the JDK code, not some magic inside JVM.
| bytes.limit(curChunkLimit) | ||
| } | ||
| // If `bytes` is an on-heap ByteBuffer, the JDK will copy it to a temporary direct | ||
| // ByteBuffer when writing it out. The JDK caches one temporary buffer per thread, and we |
There was a problem hiding this comment.
The JDK caches one temporary buffer per thread
I don't think this statement is correct. According to Util.java, the cached number of temporary direct buffer is up to IOUtil.IOV_MAX.
if the cached temp buffer gets created and freed frequently
The problem is that the varied-sized heap buffers could cause a new allocation of temporary direct buffer and free of old direct buffer if the buffer size is larger than before
|
Test build #90686 has finished for PR 21327 at commit
|
|
Test build #90689 has finished for PR 21327 at commit
|
|
LGTM. Thanks for these changes; they really help to clarify this tricky piece of code for readers. |
|
retest this please |
|
Test build #90710 has finished for PR 21327 at commit
|
|
LGTM. It's good comment added. |
|
retest this please. |
|
Test build #90715 has finished for PR 21327 at commit
|
|
thanks, merging to master! |
5 participants
What changes were proposed in this pull request?
According to the discussion in #21175 , this PR proposes 2 improvements:
limitto write outByteBufferwith slices.try ... finallyHow was this patch tested?
existing tests