Skip to content

Commit 1359c13

Browse files
LuciferYangyaooqinn
authored andcommitted
[SPARK-45776][CORE] Remove the defensive null check for MapOutputTrackerMaster#unregisterShuffle added in SPARK-39553
### What changes were proposed in this pull request? This pr Remove the defensive null check for `MapOutputTrackerMaster#unregisterShuffle` added in SPARK-39553. ### Why are the changes needed? scala/bug#12613 has been fixed in Scala 2.13.9. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing test like `SPARK-39553: Multi-thread unregister shuffle shouldn't throw NPE` in `MapOutputTrackerSuite` ### Was this patch authored or co-authored using generative AI tooling? No Closes #43644 from LuciferYang/remove-39553-null-check. Lead-authored-by: yangjie01 <[email protected]> Co-authored-by: YangJie <[email protected]> Signed-off-by: Kent Yao <[email protected]>
1 parent 1649b25 commit 1359c13

File tree

1 file changed

+2
-6
lines changed

1 file changed

+2
-6
lines changed

core/src/main/scala/org/apache/spark/MapOutputTracker.scala

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -906,12 +906,8 @@ private[spark] class MapOutputTrackerMaster(
906906
/** Unregister shuffle data */
907907
def unregisterShuffle(shuffleId: Int): Unit = {
908908
shuffleStatuses.remove(shuffleId).foreach { shuffleStatus =>
909-
// SPARK-39553: Add protection for Scala 2.13 due to https://github.com/scala/bug/issues/12613
910-
// We should revert this if Scala 2.13 solves this issue.
911-
if (shuffleStatus != null) {
912-
shuffleStatus.invalidateSerializedMapOutputStatusCache()
913-
shuffleStatus.invalidateSerializedMergeOutputStatusCache()
914-
}
909+
shuffleStatus.invalidateSerializedMapOutputStatusCache()
910+
shuffleStatus.invalidateSerializedMergeOutputStatusCache()
915911
}
916912
}
917913

0 commit comments

Comments
 (0)