[SPARK-7736] [core] [yarn] Make pyspark fail YARN app on failure. #7751

vanzin · 2015-07-29T16:35:38Z

The YARN backend doesn't like when user code calls System.exit,
since it cannot know the exit status and thus cannot set an
appropriate final status for the application.

So, for pyspark, avoid that call and instead throw an exception with
the exit code. SparkSubmit handles that exception and exits with
the given exit code, while YARN uses the exit code as the failure
code for the Spark app.

The YARN backend doesn't like when user code calls `System.exit`, since it cannot know the exit status and thus cannot set an appropriate final status for the application. So, for pyspark, avoid that call and instead throw an exception with the exit code. SparkSubmit handles that exception and exits with the given exit code, while YARN uses the exit code as the failure code for the Spark app.

vanzin · 2015-07-29T16:36:03Z

Tested on real cluster. Not sure if it's worth it to add another test in YarnClusterSuite, since those are generally slow, but it would be easy to do so.

SparkQA · 2015-07-29T19:37:45Z

Test build #38854 timed out for PR 7751 at commit 2bb2a8a after a configured wait of 175m.

vanzin · 2015-07-29T21:24:38Z

Jenkins retest this please.

andrewor14 · 2015-07-29T21:27:13Z

can you make this [Core] [YARN] so our PR dashboard displays this properly?

vanzin · 2015-07-29T22:31:27Z

Just ran into an issue where py4j threads are not daemon threads, so the YARN app is not exiting. Taking a look...

py4j uses non-daemon threads internally, so if it's not explicitly stopped, it will prevent the process from exiting now that System.exit() is not being used.

SparkQA · 2015-07-30T00:21:49Z

Test build #146 timed out for PR 7751 at commit 2bb2a8a after a configured wait of 175m.

SparkQA · 2015-07-30T00:24:21Z

Test build #38885 timed out for PR 7751 at commit 2bb2a8a after a configured wait of 175m.

SparkQA · 2015-07-30T01:53:19Z

Test build #38911 has finished for PR 7751 at commit 8df6b09.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2015-07-30T10:33:46Z

Seems pretty reasonable to me.

Conflicts: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

SparkQA · 2015-08-05T00:09:43Z

Test build #39774 has finished for PR 7751 at commit d6667da.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2015-08-05T17:48:52Z

retest this please

SparkQA · 2015-08-05T20:25:35Z

Test build #39886 has finished for PR 7751 at commit d6667da.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-05T20:27:01Z

Test build #235 has finished for PR 7751 at commit d6667da.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class BoundPortsResponse(rpcEndpointPort: Int, webUIPort: Int, restPort: Option[Int])
- ParamDesc[Array[Double]]("thresholds", "Thresholds in multi-class classification" +
- final val thresholds: DoubleArrayParam = new DoubleArrayParam(this, "thresholds", "Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values >= 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class' threshold.", (t: Array[Double]) => t.forall(_ >= 0))

vanzin · 2015-08-11T17:26:17Z

retest this please

vanzin · 2015-08-11T17:26:27Z

I'm gonna merge this later today if there are no more comments.

vanzin · 2015-08-11T23:34:19Z

retest this please

SparkQA · 2015-08-12T02:39:51Z

Test build #40540 timed out for PR 7751 at commit d6667da after a configured wait of 175m.

vanzin · 2015-08-12T02:50:10Z

retest this please

SparkQA · 2015-08-12T05:43:10Z

Test build #40576 has finished for PR 7751 at commit d6667da.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2015-08-12T07:51:05Z

retest this please

SparkQA · 2015-08-12T11:49:54Z

Test build #1494 has finished for PR 7751 at commit d6667da.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2015-08-12T16:10:00Z

This has passed tests already, all latest failures are due to flakiness...

retest this please

vanzin · 2015-08-12T22:32:11Z

retest this please

vanzin · 2015-08-12T23:02:40Z

jenkins, retest this please

SparkQA · 2015-08-13T01:38:30Z

Test build #40689 has finished for PR 7751 at commit d6667da.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2015-08-13T02:00:57Z

retest this please

SparkQA · 2015-08-13T05:04:25Z

Test build #40714 timed out for PR 7751 at commit d6667da after a configured wait of 175m.

vanzin · 2015-08-13T20:23:38Z

retest this please

SparkQA · 2015-08-13T23:24:21Z

Test build #40797 timed out for PR 7751 at commit d6667da after a configured wait of 175m.

vanzin · 2015-08-14T01:24:13Z

retest this please

SparkQA · 2015-08-14T04:29:46Z

Test build #40836 timed out for PR 7751 at commit d6667da after a configured wait of 175m.

vanzin · 2015-08-14T18:49:39Z

retest this please

vanzin · 2015-08-14T18:54:06Z

wtf

vanzin · 2015-08-14T18:54:10Z

retest this please

SparkQA · 2015-08-14T21:42:47Z

Test build #1605 has finished for PR 7751 at commit d6667da.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2015-08-14T23:15:06Z

Streaming tests are really flaky... anyway, all interesting tests passed, so unless I hear otherwise, I'll merge this Monday morning.

The YARN backend doesn't like when user code calls `System.exit`, since it cannot know the exit status and thus cannot set an appropriate final status for the application. So, for pyspark, avoid that call and instead throw an exception with the exit code. SparkSubmit handles that exception and exits with the given exit code, while YARN uses the exit code as the failure code for the Spark app. Author: Marcelo Vanzin <[email protected]> Closes #7751 from vanzin/SPARK-9416. (cherry picked from commit f68d024)

The YARN backend doesn't like when user code calls `System.exit`, since it cannot know the exit status and thus cannot set an appropriate final status for the application. So, for pyspark, avoid that call and instead throw an exception with the exit code. SparkSubmit handles that exception and exits with the given exit code, while YARN uses the exit code as the failure code for the Spark app. Author: Marcelo Vanzin <[email protected]> Closes apache#7751 from vanzin/SPARK-9416. (cherry picked from commit f68d024)

vanzin changed the title ~~[SPARK-9416] [core,yarn] Make pyspark fail YARN app on failure.~~ [SPARK-9416] [core] [yarn] Make pyspark fail YARN app on failure. Jul 29, 2015

Make sure py4j gateway is stopped.

8df6b09

py4j uses non-daemon threads internally, so if it's not explicitly stopped, it will prevent the process from exiting now that System.exit() is not being used.

vanzin changed the title ~~[SPARK-9416] [core] [yarn] Make pyspark fail YARN app on failure.~~ [SPARK-7736] [core] [yarn] Make pyspark fail YARN app on failure. Aug 3, 2015

Merge branch 'master' into SPARK-9416

d6667da

Conflicts: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

asfgit closed this in f68d024 Aug 17, 2015

vanzin deleted the SPARK-9416 branch August 17, 2015 22:22

vanzin mentioned this pull request Aug 18, 2015

[SPARK-7736] [core] Fix a race introduced in PythonRunner. #8258

Closed

[SPARK-7736] [core] [yarn] Make pyspark fail YARN app on failure. #7751

[SPARK-7736] [core] [yarn] Make pyspark fail YARN app on failure. #7751

Uh oh!

Conversation

vanzin commented Jul 29, 2015

Uh oh!

vanzin commented Jul 29, 2015

Uh oh!

SparkQA commented Jul 29, 2015

Uh oh!

vanzin commented Jul 29, 2015

Uh oh!

andrewor14 commented Jul 29, 2015

Uh oh!

vanzin commented Jul 29, 2015

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!

srowen commented Jul 30, 2015

Uh oh!

SparkQA commented Aug 5, 2015

Uh oh!

vanzin commented Aug 5, 2015

Uh oh!

SparkQA commented Aug 5, 2015

Uh oh!

SparkQA commented Aug 5, 2015

Uh oh!

vanzin commented Aug 11, 2015

Uh oh!

vanzin commented Aug 11, 2015

Uh oh!

vanzin commented Aug 11, 2015

Uh oh!

SparkQA commented Aug 12, 2015

Uh oh!

vanzin commented Aug 12, 2015

Uh oh!

SparkQA commented Aug 12, 2015

Uh oh!

andrewor14 commented Aug 12, 2015

Uh oh!

SparkQA commented Aug 12, 2015

Uh oh!

vanzin commented Aug 12, 2015

Uh oh!

vanzin commented Aug 12, 2015

Uh oh!

vanzin commented Aug 12, 2015

Uh oh!

SparkQA commented Aug 13, 2015

Uh oh!

andrewor14 commented Aug 13, 2015

Uh oh!

SparkQA commented Aug 13, 2015

Uh oh!

vanzin commented Aug 13, 2015

Uh oh!

SparkQA commented Aug 13, 2015

Uh oh!

vanzin commented Aug 14, 2015

Uh oh!

SparkQA commented Aug 14, 2015

Uh oh!

vanzin commented Aug 14, 2015

Uh oh!

vanzin commented Aug 14, 2015

Uh oh!

vanzin commented Aug 14, 2015

Uh oh!

SparkQA commented Aug 14, 2015

Uh oh!

vanzin commented Aug 14, 2015

Uh oh!

Reviewers

Assignees