[SPARK-24564][TEST] Add test suite for RecordBinaryComparator #21570
Conversation
|
Test build #91872 has finished for PR 21570 at commit
|
| private long relativeOffset(int numFields) { | ||
| // All the UnsafeRows in this suite contains less than 64 columns, so the bitSetSize shall | ||
| // always be 8. | ||
| return 8 + numFields * Long.BYTES; |
There was a problem hiding this comment.
nit: Is the multiplication factor related to Long.BYTES? It is just 8L as computeSizeInBytes uses.
|
|
||
| UnsafeRow row1 = new UnsafeRow(numFields); | ||
| byte[] data1 = new byte[100]; | ||
| row1.pointTo(data1, computeSizeInBytes(numFields * Double.BYTES)); |
There was a problem hiding this comment.
nit: ditto. Regardless of data type, width is always 8.
| UnsafeRow row1 = new UnsafeRow(numFields); | ||
| byte[] data1 = new byte[100]; | ||
| UTF8String str1 = UTF8String.fromString("Milk tea"); | ||
| row1.pointTo(data1, computeSizeInBytes(numFields * Long.BYTES + str1.numBytes())); |
There was a problem hiding this comment.
nit: ditto. Regardless of data type, width is always 8.
|
@kiszk Thanks, updated! |
| private long relativeOffset(int numFields) { | ||
| // All the UnsafeRows in this suite contains less than 64 columns, so the bitSetSize shall | ||
| // always be 8. | ||
| return 8 + numFields * 8; |
There was a problem hiding this comment.
super nit: My suggestion is to use * 8L. Although we know numFields is less than 64, it would be good to use long multiplication to avoid possible integer overflow in general.
| package test.org.apache.spark.sql.execution.sort; | ||
|
|
||
| import org.apache.spark.SparkConf; | ||
| import org.apache.spark.memory.TaskMemoryManager; |
There was a problem hiding this comment.
Since this is duplicated, it should be deleted.
There was a problem hiding this comment.
sorry but which duplicated line do you mention?
There was a problem hiding this comment.
I am sorry. Nvm, I thought lines 21 and 23 are duplicated.
| import org.apache.spark.unsafe.memory.MemoryBlock; | ||
| import org.apache.spark.unsafe.types.UTF8String; | ||
| import org.apache.spark.util.collection.unsafe.sort.*; | ||
| import org.junit.After; |
There was a problem hiding this comment.
nit: is it better to insert an empty line between L32 and L33?
|
Test build #91890 has finished for PR 21570 at commit
|
|
Test build #91887 has finished for PR 21570 at commit
|
|
retest this please |
|
Test build #91910 has finished for PR 21570 at commit
|
| taskMemoryManager.releaseExecutionMemory(size, this); | ||
| } | ||
|
|
||
| // Exposed for testing |
There was a problem hiding this comment.
super nit: we need this comment? This class itself is used for test use only?
There was a problem hiding this comment.
How about using @VisibleForTesting annotation?
| */ | ||
| public class RecordBinaryComparatorSuite { | ||
|
|
||
| private final TaskMemoryManager memoryManager = new TaskMemoryManager( |
There was a problem hiding this comment.
We need to use TaskMemoryManager for this test? Is not simple tests (e.g., just comparing unsafe rows) enough?
There was a problem hiding this comment.
Both way shall work. I choose to implement it in low level so the result won't be affected by the UnsafeProjection code.
|
Test build #92090 has finished for PR 21570 at commit
|
|
ping @JoshRosen |
|
LGTM |
| taskMemoryManager.releaseExecutionMemory(size, this); | ||
| } | ||
|
|
||
| @VisibleForTesting |
There was a problem hiding this comment.
it's already in the test package, we don't need this tag.
|
LGTM, merging to master! |
## What changes were proposed in this pull request? Add a new test suite to test RecordBinaryComparator. ## How was this patch tested? New test suite. Author: Xingbo Jiang <[email protected]> Closes apache#21570 from jiangxb1987/rbc-test.
6 participants
What changes were proposed in this pull request?
Add a new test suite to test RecordBinaryComparator.
How was this patch tested?
New test suite.