[SPARK-8392] RDDOperationGraph: getting cached nodes is slow #6839
Conversation
There was a problem hiding this comment.
Can be a val, and can add the initial elements straight away with just ListBuffer(cachedNodes:_*). Below you have a missing space before for, but better than a for loop, why not
_childClusters.foreach(cluster => cachedNodes ++= cluster._childNodes.filter(_.cached))
? Unless I overlook something that works.
Also, can you explain why you think this is slow to begin with? because nodes are expanded, then filtered?
Your JIRAs have been missing this so please add clearer motivation.
|
Yeah, I think expand all nodes then filter every node, is slow and cost memory. |
There was a problem hiding this comment.
Nit: "Return all the nodes"
|
/cc @andrewor14 for review. |
|
Hi @XuTingjun can you update the title to something more specific: |
|
add to whitelist |
There was a problem hiding this comment.
maybe I'm missing something, but why is this faster? You're still iterating through all the nodes in the end so the complexity doesn't change.
There was a problem hiding this comment.
I see, is it because we clone fewer nodes? AFAIK ++ on ArrayBuffer actually clones the entire thing first
There was a problem hiding this comment.
yeah, I think so.
|
Approach looks fine to me. Once you address the comments I'll merge this. |
|
Test build #35049 has finished for PR 6839 at commit
|
XuTingjun
changed the title
|
@andrewor14, I have updated the title and code, please have a look again, thanks. |
|
Test build #35078 has finished for PR 6839 at commit
|
|
Thanks, I'm merging this into master 1.4. |
```def getAllNodes: Seq[RDDOperationNode] =
{ _childNodes ++ _childClusters.flatMap(_.childNodes) }```
when the ```_childClusters``` has so many nodes, the process will hang on. I think we can improve the efficiency here.
Author: xutingjun <[email protected]>
Closes #6839 from XuTingjun/DAGImprove and squashes the following commits:
53b03ea [xutingjun] change code to more concise and easier to read
f98728b [xutingjun] fix words: node -> nodes
f87c663 [xutingjun] put the filter inside
81f9fd2 [xutingjun] put the filter inside
(cherry picked from commit e2cdb05)
Signed-off-by: Andrew Or <[email protected]>
```def getAllNodes: Seq[RDDOperationNode] =
{ _childNodes ++ _childClusters.flatMap(_.childNodes) }```
when the ```_childClusters``` has so many nodes, the process will hang on. I think we can improve the efficiency here.
Author: xutingjun <[email protected]>
Closes apache#6839 from XuTingjun/DAGImprove and squashes the following commits:
53b03ea [xutingjun] change code to more concise and easier to read
f98728b [xutingjun] fix words: node -> nodes
f87c663 [xutingjun] put the filter inside
81f9fd2 [xutingjun] put the filter inside
6 participants
def getAllNodes: Seq[RDDOperationNode] = { _childNodes ++ _childClusters.flatMap(_.childNodes) }when the
_childClustershas so many nodes, the process will hang on. I think we can improve the efficiency here.