Skip to content

Commit dac1047

Browse files
committed
Additional documentation for yarn memory overhead issue
1 parent 42c2c3d commit dac1047

File tree

1 file changed

+3
-0
lines changed

1 file changed

+3
-0
lines changed

yarn/common/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,9 @@ class YarnSparkHadoopUtil extends SparkHadoopUtil {
8585

8686
object YarnSparkHadoopUtil {
8787
// Additional memory overhead
88+
// 7% was arrived at experimentally. In the interest of minimizing memory waste while covering
89+
// the common cases. Memory overhead tends to grow with container size.
90+
8891
val MEMORY_OVERHEAD_FACTOR = 0.07
8992
val MEMORY_OVERHEAD_MIN = 384
9093

0 commit comments

Comments
 (0)