Skip to content

Commit 172647b

Browse files
committed
add memoryOverhead docs
1 parent a0ff545 commit 172647b

File tree

1 file changed

+14
-0
lines changed

1 file changed

+14
-0
lines changed

docs/running-on-yarn.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,20 @@ Most of the configs are the same for Spark on YARN as for other deployment modes
6767
The address of the Spark history server (i.e. host.com:18080). The address should not contain a scheme (http://). Defaults to not being set since the history server is an optional service. This address is given to the YARN ResourceManager when the Spark application finishes to link the application from the ResourceManager UI to the Spark history server UI.
6868
</td>
6969
</tr>
70+
<tr>
71+
<td><code>spark.yarn.executor.memoryOverhead</code></td>
72+
<td>384</code></td>
73+
<td>
74+
Additional memory overhead in mb for executor.
75+
</td>
76+
</tr>
77+
<tr>
78+
<td><code>spark.yarn.driver.memoryOverhead</code></td>
79+
<td>384</code></td>
80+
<td>
81+
Additional memory overhead in mb for driver.
82+
</td>
83+
</tr>
7084
</table>
7185

7286
By default, Spark on YARN will use a Spark jar installed locally, but the Spark JAR can also be in a world-readable location on HDFS. This allows YARN to cache it on nodes so that it doesn't need to be distributed each time an application runs. To point to a JAR on HDFS, `export SPARK_JAR=hdfs:///some/path`.

0 commit comments

Comments
 (0)