Skip to content

Commit 43a80c2

Browse files
viiryacmonkey
authored andcommitted
[SPARK-18800][SQL] Correct the assert in UnsafeKVExternalSorter which ensures array size
## What changes were proposed in this pull request? `UnsafeKVExternalSorter` uses `UnsafeInMemorySorter` to sort the records of `BytesToBytesMap` if it is given a map. Currently we use the number of keys in `BytesToBytesMap` to determine if the array used for sort is enough or not. We has an assert that ensures the size of the array is enough: `map.numKeys() <= map.getArray().size() / 2`. However, each record in the map takes two entries in the array, one is record pointer, another is key prefix. So the correct assert should be `map.numKeys() * 2 <= map.getArray().size() / 2`. ## How was this patch tested? N/A Please review http://spark.apache.org/contributing.html before opening a pull request. Author: Liang-Chi Hsieh <[email protected]> Closes apache#16232 from viirya/SPARK-18800-fix-UnsafeKVExternalSorter.
1 parent f4a3274 commit 43a80c2

File tree

1 file changed

+3
-1
lines changed

1 file changed

+3
-1
lines changed

sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,9 @@ public UnsafeKVExternalSorter(
9797
canUseRadixSort);
9898
} else {
9999
// The array will be used to do in-place sort, which require half of the space to be empty.
100-
assert(map.numKeys() <= map.getArray().size() / 2);
100+
// Note: each record in the map takes two entries in the array, one is record pointer,
101+
// another is the key prefix.
102+
assert(map.numKeys() * 2 <= map.getArray().size() / 2);
101103
// During spilling, the array in map will not be used, so we can borrow that and use it
102104
// as the underlying array for in-memory sorter (it's always large enough).
103105
// Since we will not grow the array, it's fine to pass `null` as consumer.

0 commit comments

Comments
 (0)