Skip to content

Commit e2cdb05

Browse files
XuTingjunAndrew Or
authored andcommitted
[SPARK-8392] RDDOperationGraph: getting cached nodes is slow
```def getAllNodes: Seq[RDDOperationNode] = { _childNodes ++ _childClusters.flatMap(_.childNodes) }``` when the ```_childClusters``` has so many nodes, the process will hang on. I think we can improve the efficiency here. Author: xutingjun <[email protected]> Closes #6839 from XuTingjun/DAGImprove and squashes the following commits: 53b03ea [xutingjun] change code to more concise and easier to read f98728b [xutingjun] fix words: node -> nodes f87c663 [xutingjun] put the filter inside 81f9fd2 [xutingjun] put the filter inside
1 parent 22732e1 commit e2cdb05

File tree

2 files changed

+4
-4
lines changed

2 files changed

+4
-4
lines changed

core/src/main/scala/org/apache/spark/ui/UIUtils.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -362,7 +362,7 @@ private[spark] object UIUtils extends Logging {
362362
{ g.incomingEdges.map { e => <div class="incoming-edge">{e.fromId},{e.toId}</div> } }
363363
{ g.outgoingEdges.map { e => <div class="outgoing-edge">{e.fromId},{e.toId}</div> } }
364364
{
365-
g.rootCluster.getAllNodes.filter(_.cached).map { n =>
365+
g.rootCluster.getCachedNodes.map { n =>
366366
<div class="cached-rdd">{n.id}</div>
367367
}
368368
}

core/src/main/scala/org/apache/spark/ui/scope/RDDOperationGraph.scala

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -66,9 +66,9 @@ private[ui] class RDDOperationCluster(val id: String, private var _name: String)
6666
_childClusters += childCluster
6767
}
6868

69-
/** Return all the nodes container in this cluster, including ones nested in other clusters. */
70-
def getAllNodes: Seq[RDDOperationNode] = {
71-
_childNodes ++ _childClusters.flatMap(_.childNodes)
69+
/** Return all the nodes which are cached. */
70+
def getCachedNodes: Seq[RDDOperationNode] = {
71+
_childNodes.filter(_.cached) ++ _childClusters.flatMap(_.getCachedNodes)
7272
}
7373
}
7474

0 commit comments

Comments
 (0)