Skip to content

[SPARK-17817][PySpark][FOLLOWUP] PySpark RDD Repartitioning Results in Highly Skewed Partition Sizes #15445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

[SPARK-17817][PySpark][FOLLOWUP] PySpark RDD Repartitioning Results in Highly Skewed Partition Sizes #15445

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
60e2abd
Decrease the batch size for repartition.
viirya Oct 7, 2016
be6d153
Merge remote-tracking branch 'upstream/master' into repartition-batch…
viirya Oct 12, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions python/pyspark/rdd.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2029,12 +2029,12 @@ def coalesce(self, numPartitions, shuffle=False):
[[1, 2, 3, 4, 5]]
"""
if shuffle:
# In Scala's repartition code, we will distribute elements evenly across output
# partitions. However, the RDD from Python is serialized as a single binary data,
# so the distribution fails and produces highly skewed partitions. We need to
# convert it to a RDD of java object before repartitioning.
data_java_rdd = self._to_java_object_rdd().coalesce(numPartitions, shuffle)
jrdd = self.ctx._jvm.SerDeUtil.javaToPython(data_java_rdd)
# Decrease the batch size in order to distribute evenly the elements across output
# partitions. Otherwise, repartition will possibly produce highly skewed partitions.
batchSize = min(10, self.ctx._batchSize or 1024)
ser = BatchedSerializer(PickleSerializer(), batchSize)
selfCopy = self._reserialize(ser)
jrdd = selfCopy._jrdd.coalesce(numPartitions, shuffle)
else:
jrdd = self._jrdd.coalesce(numPartitions, shuffle)
return RDD(jrdd, self.ctx, self._jrdd_deserializer)
Expand Down