Commit 2c3ef6a
committed
Actually filter out only the relevant RDDs
Prior to this commit, the changes in the PR actually demonstrate
little performance improvement under all workloads. This is because
we update all RDDInfos, rather than only the ones whose blocks are
being updated. Thus, even though the new filter logic in StorageStatus
is correct, we still iterate through all the RDD blocks every time
a task has an updated block.
This commit avoids this by only calling StorageLevel.updateRDDInfo
on the RDDs that need to be updated.1 parent 6fef86a commit 2c3ef6a
File tree
1 file changed
+4
-2
lines changed- core/src/main/scala/org/apache/spark/ui/storage
1 file changed
+4
-2
lines changedLines changed: 4 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
51 | | - | |
| 51 | + | |
52 | 52 | | |
53 | | - | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
54 | 56 | | |
55 | 57 | | |
56 | 58 | | |
| |||
0 commit comments