Skip to content

Commit 1fec7a5

Browse files
author
Ilya Ganelin
committed
Updated to add clarification for other driver properties
1 parent db47595 commit 1fec7a5

File tree

1 file changed

+13
-1
lines changed

1 file changed

+13
-1
lines changed

docs/configuration.md

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -116,7 +116,7 @@ of the most common options to set are:
116116
Amount of memory to use for the driver process, i.e. where SparkContext is initialized.
117117
(e.g. <code>512m</code>, <code>2g</code>).
118118

119-
<br /><br />Note: this setting only works in <code>cluster</code> mode (e.g. YARN deployment). In <code>client</code> mode (e.g. spark-shell), this setting has no effect. In client mode, driver memory should be configured in the run-time settings ; i.e. --driver-memory 2g.
119+
<br /><br />Note: setting this with <code>conf.set(...)</code> only works in <code>cluster</code> mode (e.g. YARN deployment). For <code>client</code> driver memory should be configured in the run-time settings ; i.e. --driver-memory 2g or within <code>conf/spark-defaults.conf</code>.
120120
</td>
121121
</tr>
122122
<tr>
@@ -137,6 +137,9 @@ of the most common options to set are:
137137
Having a high limit may cause out-of-memory errors in driver (depends on spark.driver.memory
138138
and memory overhead of objects in JVM). Setting a proper limit can protect the driver from
139139
out-of-memory errors.
140+
141+
<br /><br />Note: setting this with <code>conf.set(...)</code> only works in <code>cluster</code> mode (e.g. YARN deployment). For <code>client</code> driver memory should be configured in <code>conf/spark-defaults.conf</code>.
142+
140143
</td>
141144
</tr>
142145
<tr>
@@ -216,20 +219,27 @@ Apart from these, the following properties are also available, and may be useful
216219
<td>(none)</td>
217220
<td>
218221
A string of extra JVM options to pass to the driver. For instance, GC settings or other logging.
222+
223+
<br /><br />Note: setting this with <code>conf.set(...)</code> only works in <code>cluster</code> mode (e.g. YARN deployment). For <code>client</code> driver memory should be configured in <code>conf/spark-defaults.conf</code> or via the run-time settings (See Dynamically Loading Spark Properties).
224+
219225
</td>
220226
</tr>
221227
<tr>
222228
<td><code>spark.driver.extraClassPath</code></td>
223229
<td>(none)</td>
224230
<td>
225231
Extra classpath entries to append to the classpath of the driver.
232+
233+
<br /><br />Note: setting this with <code>conf.set(...)</code> only works in <code>cluster</code> mode (e.g. YARN deployment). For <code>client</code> driver memory should be configured in <code>conf/spark-defaults.conf</code> or via the run-time settings (See Dynamically Loading Spark Properties).
226234
</td>
227235
</tr>
228236
<tr>
229237
<td><code>spark.driver.extraLibraryPath</code></td>
230238
<td>(none)</td>
231239
<td>
232240
Set a special library path to use when launching the driver JVM.
241+
242+
<br /><br />Note: setting this with <code>conf.set(...)</code> only works in <code>cluster</code> mode (e.g. YARN deployment). For <code>client</code> driver memory should be configured in <code>conf/spark-defaults.conf</code> or via the run-time settings (See Dynamically Loading Spark Properties).
233243
</td>
234244
</tr>
235245
<tr>
@@ -239,6 +249,8 @@ Apart from these, the following properties are also available, and may be useful
239249
(Experimental) Whether to give user-added jars precedence over Spark's own jars when loading
240250
classes in the the driver. This feature can be used to mitigate conflicts between Spark's
241251
dependencies and user dependencies. It is currently an experimental feature.
252+
253+
<br /><br />Note: setting this with <code>conf.set(...)</code> only works in <code>cluster</code> mode (e.g. YARN deployment). For <code>client</code> driver memory should be configured in <code>conf/spark-defaults.conf</code> or via the run-time settings (See Dynamically Loading Spark Properties).
242254
</td>
243255
</tr>
244256
<tr>

0 commit comments

Comments
 (0)