Skip to content

Commit f9984d1

Browse files
Ngone51Jackey Lee
authored andcommitted
[SPARK-22465][CORE][FOLLOWUP] Use existing partitioner when defaultNumPartitions is equal to maxPartitioner.numPartitions
## What changes were proposed in this pull request? Followup of apache#20091. We could also use existing partitioner when defaultNumPartitions is equal to the maxPartitioner's numPartitions. ## How was this patch tested? Existed. Closes apache#23581 from Ngone51/dev-use-existing-partitioner-when-defaultNumPartitions-equalTo-MaxPartitioner#-numPartitions. Authored-by: Ngone51 <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
1 parent ef9a4d3 commit f9984d1

File tree

1 file changed

+4
-4
lines changed

1 file changed

+4
-4
lines changed

core/src/main/scala/org/apache/spark/Partitioner.scala

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -51,8 +51,8 @@ object Partitioner {
5151
*
5252
* When available, we choose the partitioner from rdds with maximum number of partitions. If this
5353
* partitioner is eligible (number of partitions within an order of maximum number of partitions
54-
* in rdds), or has partition number higher than default partitions number - we use this
55-
* partitioner.
54+
* in rdds), or has partition number higher than or equal to default partitions number - we use
55+
* this partitioner.
5656
*
5757
* Otherwise, we'll use a new HashPartitioner with the default partitions number.
5858
*
@@ -79,9 +79,9 @@ object Partitioner {
7979
}
8080

8181
// If the existing max partitioner is an eligible one, or its partitions number is larger
82-
// than the default number of partitions, use the existing partitioner.
82+
// than or equal to the default number of partitions, use the existing partitioner.
8383
if (hasMaxPartitioner.nonEmpty && (isEligiblePartitioner(hasMaxPartitioner.get, rdds) ||
84-
defaultNumPartitions < hasMaxPartitioner.get.getNumPartitions)) {
84+
defaultNumPartitions <= hasMaxPartitioner.get.getNumPartitions)) {
8585
hasMaxPartitioner.get.partitioner.get
8686
} else {
8787
new HashPartitioner(defaultNumPartitions)

0 commit comments

Comments
 (0)