Skip to content

[SPARK-2140] Enabling YARN memory overhead calculation from heap of the JVM #1906

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed

[SPARK-2140] Enabling YARN memory overhead calculation from heap of the JVM #1906

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0fb4499
[SPARK-1766] sorted functions to meet pedantic requirements
Aug 8, 2014
073ed82
[SPARK-2140] Enabling YARN memory overhead calculation from heap of t…
Aug 12, 2014
e623b78
Merge remote-tracking branch 'upstream/master'
Aug 12, 2014
a2abc51
Merge remote-tracking branch 'upstream/master'
Aug 14, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,11 +113,12 @@ class Client(clientArgs: ClientArguments, hadoopConf: Configuration, spConf: Spa
}

def calculateAMMemory(newApp: GetNewApplicationResponse) :Int = {
// It looks like -Xmx is handled in ClientBase. ?
// TODO: Need a replacement for the following code to fix -Xmx?
// val minResMemory: Int = newApp.getMinimumResourceCapability().getMemory()
// var amMemory = ((args.amMemory / minResMemory) * minResMemory) +
// ((if ((args.amMemory % minResMemory) == 0) 0 else minResMemory) -
// memoryOverhead )
val minResMemory: Int = newApp.getMinimumResourceCapability().getMemory()
var amMemory = ((args.amMemory / minResMemory) * minResMemory) +
((if ((args.amMemory % minResMemory) == 0) 0 else minResMemory) -
memoryOverhead )
args.amMemory
Copy link
Contributor

@witgo witgo Aug 12, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is always returns args.amMemory.

}

Expand Down