Skip to content

Conversation

@Ngone51
Copy link
Member

@Ngone51 Ngone51 commented Dec 30, 2020

What changes were proposed in this pull request?

This PR proposed to make worker/executor decommission signal configurable.

  • Added confs: spark.worker.decommission.signal / spark.executor.decommission.signal
  • Rename WorkerSigPWRReceived/ ExecutorSigPWRReceived to WorkerDecomSigReceived/ ExecutorDecomSigReceived

Why are the changes needed?

The current signal PWR can't work on macOS since it's not compliant with POSIX while macOS does. So the developers currently can't do end-to-end decommission test on their macOS environment.

Besides, the configuration becomes more flexible for users in case the default signal (PWR) gets conflicted with their own applications/environment.

Does this PR introduce any user-facing change?

No (it's a new API for 3.2)

How was this patch tested?

Manually tested.

@github-actions github-actions bot added the CORE label Dec 30, 2020
@Ngone51
Copy link
Member Author

Ngone51 commented Dec 30, 2020

cc @cloud-fan @dongjoon-hyun @holdenk Please take a look, thanks!

@SparkQA
Copy link

SparkQA commented Dec 30, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38103/

@SparkQA
Copy link

SparkQA commented Dec 30, 2020

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38103/

@SparkQA
Copy link

SparkQA commented Dec 30, 2020

Test build #133514 has finished for PR 30968 at commit 851d4c1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM (except the naming comment).
Thank you, @Ngone51 .

@SparkQA
Copy link

SparkQA commented Dec 31, 2020

Test build #133561 has finished for PR 30968 at commit 3afdf70.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 31, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38150/

@SparkQA
Copy link

SparkQA commented Dec 31, 2020

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38150/

@dongjoon-hyun
Copy link
Member

Merged to master for Apache Spark 3.2.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants