Skip to content

[SPARK-8606] Prevent exceptions in RDD.getPreferredLocations() from crashing DAGScheduler #7023

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 3 commits into from
Closed

[SPARK-8606] Prevent exceptions in RDD.getPreferredLocations() from crashing DAGScheduler #7023

wants to merge 3 commits into from

Conversation

@JoshRosen
Copy link
Contributor

@JoshRosen JoshRosen commented Jun 25, 2015

If RDD.getPreferredLocations() throws an exception it may crash the DAGScheduler and SparkContext. This patch addresses this by adding a try-catch block.

@JoshRosen
Copy link
Contributor Author

JoshRosen commented Jun 25, 2015

AFAIK RDD.getPartitions() can also throw exceptions, but I wasn't able to trigger a case where a buggy implementation of that method could crash the scheduler.

squito
squito reviewed Jun 25, 2015
View reviewed changes
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
Copy link
Contributor

@squito squito Jun 25, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

once #7014 gets merged we can actually check the causing exception (just a random comment since I just looked at that PR, this is fine as is for now)

Copy link
Contributor Author

@JoshRosen JoshRosen Jun 26, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to merge this PR now since we're going to need this string comparison in order for the regression test to run in backport branches.

Copy link
Contributor

@squito squito Jun 26, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh yeah, that is fine with me -- sorry I didn't mean to imply this should wait on the other PR. I was really just making a random observation, sorry if I confused things.

@squito
Copy link
Contributor

squito commented Jun 25, 2015

lgtm

@SparkQA
Copy link

SparkQA commented Jun 25, 2015

Test build #35800 has finished for PR 7023 at commit 770b169.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@JoshRosen
Copy link
Contributor Author

JoshRosen commented Jun 27, 2015

Alright, merging this to master + backport branches.

@asfgit asfgit closed this in 0b5abbf Jun 27, 2015
asfgit pushed a commit that referenced this pull request Jun 27, 2015
@JoshRosen
[SPARK-8606] Prevent exceptions in RDD.getPreferredLocations() from c…
e1bbf1a 
…rashing DAGScheduler

If `RDD.getPreferredLocations()` throws an exception it may crash the DAGScheduler and SparkContext. This patch addresses this by adding a try-catch block.

Author: Josh Rosen <[email protected]>

Closes #7023 from JoshRosen/SPARK-8606 and squashes the following commits:

770b169 [Josh Rosen] Fix getPreferredLocations() DAGScheduler crash with try block.
44a9b55 [Josh Rosen] Add test of a buggy getPartitions() method
19aa9f7 [Josh Rosen] Add (failing) regression test for getPreferredLocations() DAGScheduler crash

(cherry picked from commit 0b5abbf)
Signed-off-by: Josh Rosen <[email protected]>
asfgit pushed a commit that referenced this pull request Jun 27, 2015
@JoshRosen
[SPARK-8606] Prevent exceptions in RDD.getPreferredLocations() from c…
ac3591d 
…rashing DAGScheduler

If `RDD.getPreferredLocations()` throws an exception it may crash the DAGScheduler and SparkContext. This patch addresses this by adding a try-catch block.

Author: Josh Rosen <[email protected]>

Closes #7023 from JoshRosen/SPARK-8606 and squashes the following commits:

770b169 [Josh Rosen] Fix getPreferredLocations() DAGScheduler crash with try block.
44a9b55 [Josh Rosen] Add test of a buggy getPartitions() method
19aa9f7 [Josh Rosen] Add (failing) regression test for getPreferredLocations() DAGScheduler crash

(cherry picked from commit 0b5abbf)
Signed-off-by: Josh Rosen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@JoshRosen @squito @SparkQA