Skip to content

Commit 1dfbb40

Browse files
author
Marcelo Vanzin
committed
Add documentation for spark.yarn.jar.
1 parent bbdce05 commit 1dfbb40

File tree

1 file changed

+11
-2
lines changed

1 file changed

+11
-2
lines changed

docs/running-on-yarn.md

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -95,10 +95,19 @@ Most of the configs are the same for Spark on YARN as for other deployment modes
9595
The amount of off heap memory (in megabytes) to be allocated per driver. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc.
9696
</td>
9797
</tr>
98+
<tr>
99+
<td><code>spark.yarn.jar</code></td>
100+
<td>(none)</td>
101+
<td>
102+
The location of the Spark jar file, in case overriding the default location is desired.
103+
By default, Spark on YARN will use a Spark jar installed locally, but the Spark jar can also be
104+
in a world-readable location on HDFS. This allows YARN to cache it on nodes so that it doesn't
105+
need to be distributed each time an application runs. To point to a jar on HDFS, for example,
106+
set this configuration to "hdfs:///some/path".
107+
</td>
108+
</tr>
98109
</table>
99110

100-
By default, Spark on YARN will use a Spark jar installed locally, but the Spark JAR can also be in a world-readable location on HDFS. This allows YARN to cache it on nodes so that it doesn't need to be distributed each time an application runs. To point to a JAR on HDFS, `export SPARK_JAR=hdfs:///some/path`.
101-
102111
# Launching Spark on YARN
103112

104113
Ensure that `HADOOP_CONF_DIR` or `YARN_CONF_DIR` points to the directory which contains the (client side) configuration files for the Hadoop cluster.

0 commit comments

Comments
 (0)