[SPARK-1978] In some cases, spark-yarn does not automatically restart the failed container #921

witgo · 2014-05-30T16:04:51Z

No description provided.

AmplabJenkins · 2014-05-30T16:07:59Z

Merged build triggered.

AmplabJenkins · 2014-05-30T16:08:06Z

Merged build started.

sryza · 2014-05-30T16:14:09Z

This is already handled in ExecutorLauncher.launchReporterThread and ApplicationMaster.launchReporterThread, no?

AmplabJenkins · 2014-05-30T16:48:16Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-30T16:48:16Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15306/

AmplabJenkins · 2014-05-31T06:42:58Z

Merged build triggered.

AmplabJenkins · 2014-05-31T06:43:05Z

Merged build started.

witgo · 2014-05-31T06:49:49Z

@sryza
When yarnAllocator.getNumExecutorsFailed return value is greater than zero .
yarnAllocator.getNumExecutorsRunning < args.numExecutors is true forever .
That is to say,In this case,only expression userThread.isAlive or !driverClosed is false, ExecutorLauncher.launchReporterThread or ApplicationMaster.launchReporterThread will execute.

AmplabJenkins · 2014-05-31T07:23:54Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-31T07:23:54Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15314/

tgravescs · 2014-06-04T16:17:06Z

yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala

So I believe this is what this TODO is referring to so you can remove that TODO.

…eExecutors

AmplabJenkins · 2014-06-05T09:33:00Z

Merged build triggered.

AmplabJenkins · 2014-06-05T10:07:20Z

Merged build started.

AmplabJenkins · 2014-06-05T10:48:08Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-05T10:48:08Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15476/

tgravescs · 2014-06-05T18:09:48Z

yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala

similar here can you move this up above allocateResources

can you change this to have similar logic - allocate outside loop, then inside loop add missing and then allocate.

AmplabJenkins · 2014-06-09T02:57:50Z

Merged build triggered.

AmplabJenkins · 2014-06-09T02:57:56Z

Merged build started.

AmplabJenkins · 2014-06-09T03:38:33Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-09T03:38:33Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15551/

…eExecutors

AmplabJenkins · 2014-06-09T15:22:50Z

Merged build triggered.

AmplabJenkins · 2014-06-09T15:22:56Z

Merged build started.

AmplabJenkins · 2014-06-09T16:02:46Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-09T16:02:46Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15568/

tgravescs · 2014-06-09T18:41:27Z

Thanks @witgo if you can change the order of the logic in the ExecutorLauncher to match, this looks good.

…eExecutors

AmplabJenkins · 2014-06-10T01:42:49Z

Merged build triggered.

witgo · 2014-06-10T01:43:09Z

Done

AmplabJenkins · 2014-06-10T04:35:23Z

Merged build started.

AmplabJenkins · 2014-06-10T05:16:08Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-10T05:16:08Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15593/

tgravescs · 2014-06-10T15:36:41Z

Looks good, +1. Thanks @witgo

… the failed container Author: witgo <[email protected]> Closes #921 from witgo/allocateExecutors and squashes the following commits: bc3aa66 [witgo] review commit 8800eba [witgo] Merge branch 'master' of https://github.com/apache/spark into allocateExecutors 32ac7af [witgo] review commit 056b8c7 [witgo] Merge branch 'master' of https://github.com/apache/spark into allocateExecutors 04c6f7e [witgo] Merge branch 'master' into allocateExecutors aff827c [witgo] review commit 5c376e0 [witgo] Merge branch 'master' of https://github.com/apache/spark into allocateExecutors 1faf4f4 [witgo] Merge branch 'master' into allocateExecutors 3c464bd [witgo] add time limit to allocateExecutors e00b656 [witgo] In some cases, yarn does not automatically restart the container

tgravescs · 2014-06-10T18:44:04Z

I merged this into branch-1.0 also

… the failed container Author: witgo <[email protected]> Closes apache#921 from witgo/allocateExecutors and squashes the following commits: bc3aa66 [witgo] review commit 8800eba [witgo] Merge branch 'master' of https://github.com/apache/spark into allocateExecutors 32ac7af [witgo] review commit 056b8c7 [witgo] Merge branch 'master' of https://github.com/apache/spark into allocateExecutors 04c6f7e [witgo] Merge branch 'master' into allocateExecutors aff827c [witgo] review commit 5c376e0 [witgo] Merge branch 'master' of https://github.com/apache/spark into allocateExecutors 1faf4f4 [witgo] Merge branch 'master' into allocateExecutors 3c464bd [witgo] add time limit to allocateExecutors e00b656 [witgo] In some cases, yarn does not automatically restart the container

…IPS (apache#921)

In some cases, yarn does not automatically restart the container

e00b656

witgo changed the title ~~In some cases, yarn does not automatically restart the container~~ [WIP] In some cases, yarn does not automatically restart the container May 30, 2014

add time limit to allocateExecutors

3c464bd

witgo changed the title ~~[WIP] In some cases, yarn does not automatically restart the container~~ In some cases, yarn does not automatically restart the container May 31, 2014

witgo changed the title ~~In some cases, yarn does not automatically restart the container~~ In some cases, spark-yarn does not automatically restart the container May 31, 2014

witgo changed the title ~~In some cases, spark-yarn does not automatically restart the container~~ In some cases, spark-yarn does not automatically restart the failed container May 31, 2014

witgo changed the title ~~In some cases, spark-yarn does not automatically restart the failed container~~ [SPARK-1978] In some cases, spark-yarn does not automatically restart the failed container May 31, 2014

tgravescs reviewed Jun 4, 2014
View reviewed changes

witgo added 3 commits June 5, 2014 17:07

Merge branch 'master' into allocateExecutors

1faf4f4

Merge branch 'master' of https://github.com/apache/spark into allocat…

5c376e0

…eExecutors

review commit

aff827c

tgravescs reviewed Jun 5, 2014
View reviewed changes

witgo added 2 commits June 9, 2014 23:13

Merge branch 'master' of https://github.com/apache/spark into allocat…

056b8c7

…eExecutors

review commit

32ac7af

witgo added 2 commits June 10, 2014 09:39

Merge branch 'master' of https://github.com/apache/spark into allocat…

8800eba

…eExecutors

review commit

bc3aa66

asfgit closed this in 884ca71 Jun 10, 2014

witgo deleted the allocateExecutors branch June 10, 2014 15:38

agirish pushed a commit to HPEEzmeral/apache-spark that referenced this pull request May 5, 2022

MapR [SPARK-952] Spark services can't start on cluster with enabled F…

85b3d44

…IPS (apache#921)

udaynpusa pushed a commit to mapr/spark that referenced this pull request Jan 30, 2024

MapR [SPARK-952] Spark services can't start on cluster with enabled F…

a180c0a

…IPS (apache#921)

mapr-devops pushed a commit to mapr/spark that referenced this pull request May 8, 2025

MapR [SPARK-952] Spark services can't start on cluster with enabled F…

fbbb6f8

…IPS (apache#921)

[SPARK-1978] In some cases, spark-yarn does not automatically restart the failed container #921

[SPARK-1978] In some cases, spark-yarn does not automatically restart the failed container #921

Uh oh!

Conversation

witgo commented May 30, 2014

Uh oh!

AmplabJenkins commented May 30, 2014

Uh oh!

AmplabJenkins commented May 30, 2014

Uh oh!

sryza commented May 30, 2014

Uh oh!

AmplabJenkins commented May 30, 2014

Uh oh!

AmplabJenkins commented May 30, 2014

Uh oh!

AmplabJenkins commented May 31, 2014

Uh oh!

AmplabJenkins commented May 31, 2014

Uh oh!

witgo commented May 31, 2014

Uh oh!

AmplabJenkins commented May 31, 2014

Uh oh!

AmplabJenkins commented May 31, 2014

Uh oh!

tgravescs Jun 4, 2014

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jun 5, 2014

Uh oh!

AmplabJenkins commented Jun 5, 2014

Uh oh!

AmplabJenkins commented Jun 5, 2014

Uh oh!

AmplabJenkins commented Jun 5, 2014

Uh oh!

tgravescs Jun 5, 2014

Choose a reason for hiding this comment

Uh oh!

tgravescs Jun 9, 2014

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

tgravescs commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

witgo commented Jun 10, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

tgravescs commented Jun 10, 2014

Uh oh!

tgravescs commented Jun 10, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone