Skip to content

[SPARK-2675] Increase EVENT_QUEUE_CAPACITY by 20x. #1579

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[SPARK-2675] Increase EVENT_QUEUE_CAPACITY by 20x. #1579

wants to merge 1 commit into from

Conversation

@concretevitamin
Copy link
Contributor

@concretevitamin concretevitamin commented Jul 24, 2014

@concretevitamin
Copy link
Contributor Author

concretevitamin commented Jul 24, 2014

Jenkins, test this please.

@SparkQA
Copy link

SparkQA commented Jul 24, 2014

QA tests have started for PR 1579. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17131/consoleFull

@shivaram
Copy link
Contributor

shivaram commented Jul 24, 2014

Is there an estimate on the size of an event ? How much memory would 200000 events take ?

@SparkQA
Copy link

SparkQA commented Jul 24, 2014

QA results for PR 1579:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17131/consoleFull

@lianhuiwang
Copy link
Contributor

lianhuiwang commented Jul 25, 2014

i want to know how much memory that @shivaram said before. @concretevitamin can show the number?thanks.

@concretevitamin
Copy link
Contributor Author

concretevitamin commented Jul 25, 2014

Unfortunately I may not be able to find time to run an experiment soon. If anyone is interested, I think org.apache.spark.util.SizeEstimator is reasonable to try. Let's also see if SPARK-2316 gets fixed in the near future, which may remove the need to increase the queue capacity.

@andrewor14
Copy link
Contributor

andrewor14 commented Jul 25, 2014

However many others also face this issue (evident from the mailing list). We should aim to fix this issue one way or another, either through this PR and/or the fix for SPARK-2316, by the 1.1 release. SizeEstimator is a good place to start.

@andrewor14 andrewor14 mentioned this pull request Aug 1, 2014
Closed
@pwendell
Copy link
Contributor

pwendell commented Aug 2, 2014

@concretevitamin would you mind closing this? The underlying issue has been fixed in #1679?

@pwendell pwendell mentioned this pull request Aug 2, 2014
Closed
sunchao pushed a commit to sunchao/spark that referenced this pull request Jun 2, 2023
@kazuyukitanimura
Remove DisableBoson from JDBCV2Suite (apache#1579)
3cb5ab6 
This is another PR that removes `DisableBoson` from `JDBCV2Suite` as pie/boson#571 fixed the locale issue.

Tested in my own rio
https://rio.apple.com/projects/aci-kazuyuki-tanimura-boson/pipeline-specs/aci-kazuyuki-tanimura-boson-master-spark-sql-core-1-scan-only/pipelines/fb3190d6-ce12-4cca-b1c1-094a1e4d21fe
https://rio.apple.com/projects/aci-kazuyuki-tanimura-boson/pipeline-specs/aci-kazuyuki-tanimura-boson-master-spark-sql-core-2-scan-only/pipelines/e873fb32-db32-4ab9-b126-7b536e24c7a8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@concretevitamin @SparkQA @shivaram @lianhuiwang @andrewor14 @pwendell