Skip to content

Commit 3519b5e

Browse files
committed
[SPARK-2967][SQL] Follow-up: Also copy hash expressions in sort based shuffle fix.
Follow-up to apache#2066 Author: Michael Armbrust <[email protected]> Closes apache#2072 from marmbrus/sortShuffle and squashes the following commits: 2ff8114 [Michael Armbrust] Fix bug
1 parent 7e191fe commit 3519b5e

File tree

1 file changed

+6
-3
lines changed

1 file changed

+6
-3
lines changed

sql/core/src/main/scala/org/apache/spark/sql/execution/Exchange.scala

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -46,12 +46,15 @@ case class Exchange(newPartitioning: Partitioning, child: SparkPlan) extends Una
4646
case HashPartitioning(expressions, numPartitions) =>
4747
// TODO: Eliminate redundant expressions in grouping key and value.
4848
val rdd = child.execute().mapPartitions { iter =>
49-
@transient val hashExpressions =
50-
newMutableProjection(expressions, child.output)()
51-
5249
if (sortBasedShuffleOn) {
50+
@transient val hashExpressions =
51+
newProjection(expressions, child.output)
52+
5353
iter.map(r => (hashExpressions(r), r.copy()))
5454
} else {
55+
@transient val hashExpressions =
56+
newMutableProjection(expressions, child.output)()
57+
5558
val mutablePair = new MutablePair[Row, Row]()
5659
iter.map(r => mutablePair.update(hashExpressions(r), r))
5760
}

0 commit comments

Comments
 (0)