-
Notifications
You must be signed in to change notification settings - Fork 28.9k
[SPARK-31946][CORE] Make worker/executor decommission signal configurable #30968
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
cc @cloud-fan @dongjoon-hyun @holdenk Please take a look, thanks! |
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Test build #133514 has finished for PR 30968 at commit
|
core/src/main/scala/org/apache/spark/deploy/DeployMessage.scala
Outdated
Show resolved
Hide resolved
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedClusterMessage.scala
Outdated
Show resolved
Hide resolved
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, LGTM (except the naming comment).
Thank you, @Ngone51 .
|
Test build #133561 has finished for PR 30968 at commit
|
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Merged to master for Apache Spark 3.2.0. |
What changes were proposed in this pull request?
This PR proposed to make worker/executor decommission signal configurable.
spark.worker.decommission.signal/spark.executor.decommission.signalWorkerSigPWRReceived/ExecutorSigPWRReceivedtoWorkerDecomSigReceived/ExecutorDecomSigReceivedWhy are the changes needed?
The current signal
PWRcan't work on macOS since it's not compliant with POSIX while macOS does. So the developers currently can't do end-to-end decommission test on their macOS environment.Besides, the configuration becomes more flexible for users in case the default signal (
PWR) gets conflicted with their own applications/environment.Does this PR introduce any user-facing change?
No (it's a new API for 3.2)
How was this patch tested?
Manually tested.