[SPARK-16230] [CORE] CoarseGrainedExecutorBackend to self kill if there is an exception while creating an Executor #14202
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #62324 has finished for PR 14202 at commit
|
Contributor
Author
|
Hi @zsxwing , can you please review this diff ? |
Member
There was a problem hiding this comment.
@tejasapatil do you know if all log frameworks can handle null throwable here? Perhaps it's better to handle it by ourselves like:
if (throwable != null) {
logError(reason, throwable)
} else {
logError(reason)
}
Contributor
Author
There was a problem hiding this comment.
Ok. Did the change.
Member
|
@tejasapatil Looks pretty good. Just one minor comment. |
…ile creating an Executor
tejasapatil
force-pushed
the
exit_executor_failure
branch
from
0c71699 to
5499071
Compare
|
Test build #62381 has finished for PR 14202 at commit
|
Contributor
Author
|
@zsxwing : I am done with the change(s) you suggested. |
Member
|
LGTM. Thanks! Merging to master and 2.0. |
asfgit
pushed a commit
that referenced
this pull request
Jul 15, 2016
…e is an exception while creating an Executor ## What changes were proposed in this pull request? With the fix from SPARK-13112, I see that `LaunchTask` is always processed after `RegisteredExecutor` is done and so it gets chance to do all retries to startup an executor. There is still a problem that if `Executor` creation itself fails and there is some exception, it gets unnoticed and the executor is killed when it tries to process the `LaunchTask` as `executor` is null : https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala#L88 So if one looks at the logs, it does not tell that there was problem during `Executor` creation and thats why it was killed. This PR explicitly catches exception in `Executor` creation, logs a proper message and then exits the JVM. Also, I have changed the `exitExecutor` method to accept `reason` so that backends can use that reason and do stuff like logging to a DB to get an aggregate of such exits at a cluster level ## How was this patch tested? I am relying on existing tests Author: Tejas Patil <[email protected]> Closes #14202 from tejasapatil/exit_executor_failure. (cherry picked from commit b2f24f9) Signed-off-by: Shixiong Zhu <[email protected]>
zzcclp
added a commit
to zzcclp/spark
that referenced
this pull request
Jul 18, 2016
…f there is an exception while creating an Executor apache#14202
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
With the fix from SPARK-13112, I see that
LaunchTaskis always processed afterRegisteredExecutoris done and so it gets chance to do all retries to startup an executor. There is still a problem that ifExecutorcreation itself fails and there is some exception, it gets unnoticed and the executor is killed when it tries to process theLaunchTaskasexecutoris null : https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala#L88 So if one looks at the logs, it does not tell that there was problem duringExecutorcreation and thats why it was killed.This PR explicitly catches exception in
Executorcreation, logs a proper message and then exits the JVM. Also, I have changed theexitExecutormethod to acceptreasonso that backends can use that reason and do stuff like logging to a DB to get an aggregate of such exits at a cluster levelHow was this patch tested?
I am relying on existing tests