[SPARK-12081] Make unified memory manager work with small heaps #10081
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The new space used by storage and execution will be calculated by (JVM size - 300MB) * 75%, the `spark.memory.fraction`.
Contributor
Author
Contributor
|
LGTM |
Contributor
There was a problem hiding this comment.
you should document what this is and update the classdoc for UnifiedMemoryManager
|
Test build #2141 has started for PR 10081 at commit |
|
Test build #2140 has started for PR 10081 at commit |
|
Test build #2139 has finished for PR 10081 at commit
|
|
Test build #47025 has finished for PR 10081 at commit
|
Contributor
Author
|
Thanks I'm merging into master 1.6 |
asfgit
pushed a commit
that referenced
this pull request
Dec 2, 2015
The existing `spark.memory.fraction` (default 0.75) gives the system 25% of the space to work with. For small heaps, this is not enough: e.g. default 1GB leaves only 250MB system memory. This is especially a problem in local mode, where the driver and executor are crammed in the same JVM. Members of the community have reported driver OOM's in such cases. **New proposal.** We now reserve 300MB before taking the 75%. For 1GB JVMs, this leaves `(1024 - 300) * 0.75 = 543MB` for execution and storage. This is proposal (1) listed in the [JIRA](https://issues.apache.org/jira/browse/SPARK-12081). Author: Andrew Or <[email protected]> Closes #10081 from andrewor14/unified-memory-small-heaps. (cherry picked from commit d96f8c9) Signed-off-by: Andrew Or <[email protected]>
|
Test build #47014 has finished for PR 10081 at commit
|
Contributor
There was a problem hiding this comment.
For non-small heap, the above memory doesn't need to be reserved, right ?
| val minSystemMemory = reservedMemory * 1.5 | ||
| if (systemMemory < minSystemMemory) { | ||
| throw new IllegalArgumentException(s"System memory $systemMemory must " + | ||
| s"be at least $minSystemMemory. Please use a larger heap size.") |
There was a problem hiding this comment.
This comes out as scientific notation, which made me laugh.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The existing
spark.memory.fraction(default 0.75) gives the system 25% of the space to work with. For small heaps, this is not enough: e.g. default 1GB leaves only 250MB system memory. This is especially a problem in local mode, where the driver and executor are crammed in the same JVM. Members of the community have reported driver OOM's in such cases.New proposal. We now reserve 300MB before taking the 75%. For 1GB JVMs, this leaves
(1024 - 300) * 0.75 = 543MBfor execution and storage. This is proposal (1) listed in the JIRA.