This repository was archived by the owner on Jan 9, 2020. It is now read-only.
Set ENV_DRIVER_MEMORY to memory instead of memory+overhead #475
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
duyanghao
force-pushed
the
fix_driverJvmMemory_branch
branch
from
06db577 to
332b3e8
Compare
This was referenced Aug 31, 2017
duyanghao
force-pushed
the
fix_driverJvmMemory_branch
branch
3 times, most recently
from
61cf580 to
c7179b4
Compare
Signed-off-by: duyanghao <[email protected]>
duyanghao
force-pushed
the
fix_driverJvmMemory_branch
branch
from
c7179b4 to
cab62f2
Compare
ash211
reviewed
Aug 31, 2017
ash211
left a comment
ash211
left a comment
There was a problem hiding this comment.
Looks right to me, please fix the test.
Thanks for spotting this @duyanghao ! I completely missed it when I was just in here
| .toMap | ||
| assert(envs.size === 6) | ||
| assert(envs(ENV_SUBMIT_EXTRA_CLASSPATH) === "/opt/spark/spark-exmaples.jar") | ||
| assert(envs(ENV_DRIVER_MEMORY) === "456M") |
There was a problem hiding this comment.
please change the value here of what this should now be equal to in this test
|
Pushed a change that restored the test -- will merge with an approval from @mccheah |
mccheah
approved these changes
Aug 31, 2017
ifilonenko
pushed a commit
to ifilonenko/spark
that referenced
this pull request
Feb 26, 2019
https://issues.apache.org/jira/browse/SPARK-26626 apache#23556 ## What changes were proposed in this pull request? This adds a `spark.sql.maxRepeatedAliasSize` config option, which specifies the maximum size of an aliased expression to be substituted (in CollapseProject and PhysicalOperation). This prevents large aliased expressions from being substituted multiple times and exploding the size of the expression tree, eventually OOMing the driver. The default config value of 100 was chosen through testing to find the optimally performant value:  ## How was this patch tested? Added unit tests, and did manual testing
puneetloya
pushed a commit
to puneetloya/spark
that referenced
this pull request
Mar 11, 2019
…ark-on-k8s#475) * Set ENV_DRIVER_MEMORY to memory instead of memory+overhead Signed-off-by: duyanghao <[email protected]> * Restore test
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Fix #471(Set
ENV_DRIVER_MEMORYto memory instead of memory+overhead)How was this patch tested?
Manual tests show settings do work.