Skip to content

Modify default YARN memory_overhead-- from an additive constant to a multiplier #1391

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 17 commits into from
Closed

Modify default YARN memory_overhead-- from an additive constant to a multiplier #1391

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
681b36f
Fix for SPARK-1758: failing test org.apache.spark.JavaAPISuite.wholeT…
nishkamravi2 May 8, 2014
5108700
Fix in Spark for the Concurrent thread modification issue (SPARK-1097…
nishkamravi2 Jun 3, 2014
6b840f0
Undo the fix for SPARK-1758 (the problem is fixed)
nishkamravi2 Jun 3, 2014
df2aeb1
Improved fix for ConcurrentModificationIssue (Spark-1097, Hadoop-10456)
nishkamravi2 Jun 9, 2014
eb663ca
Merge branch 'master' of https://github.com/apache/spark
nishkamravi2 Jun 9, 2014
5423a03
Merge branch 'master' of https://github.com/apache/spark
nishkamravi2 Jun 10, 2014
3bf8fad
Merge branch 'master' of https://github.com/apache/spark
nishkamravi2 Jun 16, 2014
2b630f9
Accept memory input as "30g", "512M" instead of an int value, to be c…
nishkamravi2 Jun 16, 2014
efd688a
Merge branch 'master' of https://github.com/apache/spark
nishkamravi2 Jul 13, 2014
715201e
Modify default YARN memory_overhead: from an additive constant to a m…
nishkamravi2 Jul 13, 2014
11a231c
Update running-on-yarn.md
nishkamravi2 Jul 17, 2014
b44845b
Update YarnAllocationHandler.scala
nishkamravi2 Jul 17, 2014
44ba4da
Update YarnAllocationHandler.scala
nishkamravi2 Jul 17, 2014
3062d6b
Update ExecutorLauncher.scala
nishkamravi2 Sep 19, 2014
b8eeefe
Update YarnAllocationHandler.scala
nishkamravi2 Sep 19, 2014
492c33f
Update ClientBase.scala
nishkamravi2 Sep 19, 2014
5cbe89c
Update YarnAllocationHandler.scala
nishkamravi2 Sep 19, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/running-on-yarn.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,14 +83,14 @@ Most of the configs are the same for Spark on YARN as for other deployment modes
</tr>
<tr>
<td><code>spark.yarn.executor.memoryOverhead</code></td>
<td>384</td>
<td>executorMemory * 0.06, with minimum of 384 </td>
<td>
The amount of off heap memory (in megabytes) to be allocated per executor. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc.
</td>
</tr>
<tr>
<td><code>spark.yarn.driver.memoryOverhead</code></td>
<td>384</td>
<td>driverMemory * 0.06, with minimum of 384 </td>
<td>
The amount of off heap memory (in megabytes) to be allocated per driver. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc.
</td>
Expand Down
3 changes: 2 additions & 1 deletion yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,8 @@ class ExecutorLauncher(args: ApplicationMasterArguments, conf: Configuration, sp

if (minimumMemory > 0) {
val mem = args.executorMemory + sparkConf.getInt("spark.yarn.executor.memoryOverhead",
YarnAllocationHandler.MEMORY_OVERHEAD)
math.max((YarnAllocationHandler.MEMORY_OVERHEAD_FACTOR * args.executorMemory).toInt,
YarnAllocationHandler.MEMORY_OVERHEAD_MIN))
val numCore = (mem / minimumMemory) + (if (0 != (mem % minimumMemory)) 1 else 0)

if (numCore > 0) {
Expand Down
8 changes: 5 additions & 3 deletions yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,8 @@ private[yarn] class YarnAllocationHandler(

// Additional memory overhead - in mb.
private def memoryOverhead: Int = sparkConf.getInt("spark.yarn.executor.memoryOverhead",
YarnAllocationHandler.MEMORY_OVERHEAD)
math.max((YarnAllocationHandler.MEMORY_OVERHEAD_FACTOR * executorMemory).toInt,
YarnAllocationHandler.MEMORY_OVERHEAD_MIN))

private val numExecutorsRunning = new AtomicInteger()
// Used to generate a unique id per executor
Expand Down Expand Up @@ -548,8 +549,9 @@ object YarnAllocationHandler {
// request types (like map/reduce in hadoop for example)
val PRIORITY = 1

// Additional memory overhead - in mb
val MEMORY_OVERHEAD = 384
// Additional memory overhead
val MEMORY_OVERHEAD_FACTOR = 0.06
val MEMORY_OVERHEAD_MIN = 384

// Host to rack map - saved from allocation requests
// We are expecting this not to change.
Expand Down
3 changes: 2 additions & 1 deletion yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,8 @@ trait ClientBase extends Logging {

// Additional memory overhead - in mb.
protected def memoryOverhead: Int = sparkConf.getInt("spark.yarn.driver.memoryOverhead",
YarnAllocationHandler.MEMORY_OVERHEAD)
math.max((YarnAllocationHandler.MEMORY_OVERHEAD_FACTOR * args.amMemory).toInt,
YarnAllocationHandler.MEMORY_OVERHEAD_MIN))

// TODO(harvey): This could just go in ClientArguments.
def validateArgs() = {
Expand Down
8 changes: 5 additions & 3 deletions yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,8 @@ private[yarn] class YarnAllocationHandler(

// Additional memory overhead - in mb.
private def memoryOverhead: Int = sparkConf.getInt("spark.yarn.executor.memoryOverhead",
YarnAllocationHandler.MEMORY_OVERHEAD)
math.max((YarnAllocationHandler.MEMORY_OVERHEAD_FACTOR * executorMemory).toInt,
YarnAllocationHandler.MEMORY_OVERHEAD_MIN))

// Number of container requests that have been sent to, but not yet allocated by the
// ApplicationMaster.
Expand Down Expand Up @@ -562,8 +563,9 @@ object YarnAllocationHandler {
// request types (like map/reduce in hadoop for example)
val PRIORITY = 1

// Additional memory overhead - in mb.
val MEMORY_OVERHEAD = 384
// Additional memory overhead
val MEMORY_OVERHEAD_FACTOR = 0.06
val MEMORY_OVERHEAD_MIN = 384

// Host to rack map - saved from allocation requests. We are expecting this not to change.
// Note that it is possible for this to change : and ResurceManager will indicate that to us via
Expand Down