Skip to content

Commit 9b4da7b

Browse files
SereneAntsrowen
authored andcommitted
[SPARK-21491][GRAPHX] Enhance GraphX performance: breakOut instead of .toMap
## What changes were proposed in this pull request? `Traversable.toMap` changed to 'collections.breakOut', that eliminates intermediate tuple collection creation, see [Stack Overflow article](https://stackoverflow.com/questions/1715681/scala-2-8-breakout). ## How was this patch tested? Unit tests run. No performance tests performed yet. Please review http://spark.apache.org/contributing.html before opening a pull request. Author: iurii.ant <[email protected]> Closes #18693 from SereneAnt/performance_toMap-breakOut.
1 parent 06a9793 commit 9b4da7b

File tree

2 files changed

+4
-3
lines changed

2 files changed

+4
-3
lines changed

graphx/src/main/scala/org/apache/spark/graphx/lib/LabelPropagation.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ object LabelPropagation {
5555
val count1Val = count1.getOrElse(i, 0L)
5656
val count2Val = count2.getOrElse(i, 0L)
5757
i -> (count1Val + count2Val)
58-
}.toMap
58+
}(collection.breakOut) // more efficient alternative to [[collection.Traversable.toMap]]
5959
}
6060
def vertexProgram(vid: VertexId, attr: Long, message: Map[VertexId, Long]): VertexId = {
6161
if (message.isEmpty) attr else message.maxBy(_._2)._1

graphx/src/main/scala/org/apache/spark/graphx/lib/ShortestPaths.scala

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,10 +33,11 @@ object ShortestPaths {
3333

3434
private def incrementMap(spmap: SPMap): SPMap = spmap.map { case (v, d) => v -> (d + 1) }
3535

36-
private def addMaps(spmap1: SPMap, spmap2: SPMap): SPMap =
36+
private def addMaps(spmap1: SPMap, spmap2: SPMap): SPMap = {
3737
(spmap1.keySet ++ spmap2.keySet).map {
3838
k => k -> math.min(spmap1.getOrElse(k, Int.MaxValue), spmap2.getOrElse(k, Int.MaxValue))
39-
}.toMap
39+
}(collection.breakOut) // more efficient alternative to [[collection.Traversable.toMap]]
40+
}
4041

4142
/**
4243
* Computes shortest paths to the given set of landmark vertices.

0 commit comments

Comments
 (0)