Skip to content

[SPARK-2898] [PySpark] fix bugs in deamon.py #1842

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 4 commits into from
Closed

[SPARK-2898] [PySpark] fix bugs in deamon.py #1842

wants to merge 4 commits into from

Conversation

@davies
Copy link
Contributor

@davies davies commented Aug 7, 2014

  1. do not use signal handler for SIGCHILD, it's easy to cause deadlock
  2. handle EINTR during accept()
  3. pass errno into JVM
  4. handle EAGAIN during fork()

Now, it can pass 50k tasks tests in 180 seconds.

@davies
fix bugs in deamon.py
0cd0817 
1. do not use signal handler for SIGCHILD, it's easy to cause deadlock
2. handle EINTR during accept()
3. pass errno into JVM
4. handle EAGAIN during fork()

Now, it can pass 50k tasks tests in 180 seconds.
@SparkQA
Copy link

SparkQA commented Aug 7, 2014

QA tests have started for PR 1842. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18146/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 7, 2014

QA results for PR 1842:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18146/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 7, 2014

QA tests have started for PR 1842. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18148/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 7, 2014

QA results for PR 1842:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18148/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 7, 2014

QA tests have started for PR 1842. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18153/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 7, 2014

QA results for PR 1842:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18153/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 8, 2014

QA tests have started for PR 1842. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18158/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 8, 2014

QA results for PR 1842:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18158/consoleFull

@JoshRosen
Copy link
Contributor

JoshRosen commented Aug 8, 2014

This looks like a good collection of fixes. Let me give it one final look tomorrow morning, then I'll merge it.

@JoshRosen
Copy link
Contributor

JoshRosen commented Aug 10, 2014

I've merged this into master and branch-1.1. Thanks!

asfgit pushed a commit that referenced this pull request Aug 10, 2014
@davies @JoshRosen
[SPARK-2898] [PySpark] fix bugs in deamon.py
92daffe 
1. do not use signal handler for SIGCHILD, it's easy to cause deadlock
2. handle EINTR during accept()
3. pass errno into JVM
4. handle EAGAIN during fork()

Now, it can pass 50k tasks tests in 180 seconds.

Author: Davies Liu <[email protected]>

Closes #1842 from davies/qa and squashes the following commits:

f0ea451 [Davies Liu] fix lint
03a2e8c [Davies Liu] cleanup dead children every seconds
32cb829 [Davies Liu] fix lint
0cd0817 [Davies Liu] fix bugs in deamon.py

(cherry picked from commit 28dcbb5)
Signed-off-by: Josh Rosen <[email protected]>
@asfgit asfgit closed this in 28dcbb5 Aug 10, 2014
xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@davies @conviva-zz
[SPARK-2898] [PySpark] fix bugs in deamon.py
2694f80 
1. do not use signal handler for SIGCHILD, it's easy to cause deadlock
2. handle EINTR during accept()
3. pass errno into JVM
4. handle EAGAIN during fork()

Now, it can pass 50k tasks tests in 180 seconds.

Author: Davies Liu <[email protected]>

Closes apache#1842 from davies/qa and squashes the following commits:

f0ea451 [Davies Liu] fix lint
03a2e8c [Davies Liu] cleanup dead children every seconds
32cb829 [Davies Liu] fix lint
0cd0817 [Davies Liu] fix bugs in deamon.py
@davies davies deleted the qa branch September 15, 2014 22:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@davies @SparkQA @JoshRosen