You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/configuration.md
+17-8Lines changed: 17 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -116,8 +116,10 @@ of the most common options to set are:
116
116
Amount of memory to use for the driver process, i.e. where SparkContext is initialized.
117
117
(e.g. <code>512m</code>, <code>2g</code>).
118
118
119
-
<br /><em>Note:</em> setting this with <code>conf.set(...)</code> only works in <code>cluster</code> mode (e.g. YARN deployment). For <code>client</code> driver memory should be configured in the run-time settings ; i.e. --driver-memory 2g or within <code>conf/spark-defaults.conf</code>.
120
-
</td>
119
+
<br /><em>Note:</em> In client mode, this config must not be set through the <code>SparkConf</code>
120
+
directly in your application, because the driver JVM has already started at that point.
121
+
Instead, please set this through the <code>--driver-memory</code> command line option
122
+
or in your default properties file.</td>
121
123
</tr>
122
124
<tr>
123
125
<td><code>spark.executor.memory</code></td>
@@ -138,8 +140,9 @@ of the most common options to set are:
138
140
and memory overhead of objects in JVM). Setting a proper limit can protect the driver from
139
141
out-of-memory errors.
140
142
141
-
<br /><em>Note:</em> setting this with <code>conf.set(...)</code> only works in <code>cluster</code> mode (e.g. YARN deployment). For <code>client</code> driver memory should be configured in <code>conf/spark-defaults.conf</code>.
142
-
143
+
<br /><em>Note:</em> In client mode, this config must not be set through the <code>SparkConf</code>
144
+
directly in your application, because the driver JVM has already started at that point.
145
+
Instead, please set this through the default properties file.</td>
143
146
</td>
144
147
</tr>
145
148
<tr>
@@ -220,8 +223,10 @@ Apart from these, the following properties are also available, and may be useful
220
223
<td>
221
224
A string of extra JVM options to pass to the driver. For instance, GC settings or other logging.
222
225
223
-
<br /><em>Note:</em> setting this with <code>conf.set(...)</code> only works in <code>cluster</code> mode (e.g. YARN deployment). For <code>client</code> driver memory should be configured in <code>conf/spark-defaults.conf</code> or via the run-time settings (See Dynamically Loading Spark Properties).
224
-
226
+
<br /><em>Note:</em> In client mode, this config must not be set through the <code>SparkConf</code>
227
+
directly in your application, because the driver JVM has already started at that point.
228
+
Instead, please set this through the command line option (see Dynamically Loading Spark Properties)
229
+
or in your default properties file.</td>
225
230
</td>
226
231
</tr>
227
232
<tr>
@@ -230,7 +235,9 @@ Apart from these, the following properties are also available, and may be useful
230
235
<td>
231
236
Extra classpath entries to append to the classpath of the driver.
232
237
233
-
<br /><em>Note:</em> setting this with <code>conf.set(...)</code> only works in <code>cluster</code> mode (e.g. YARN deployment). For <code>client</code> driver memory should be configured in <code>conf/spark-defaults.conf</code> or via the run-time settings (See Dynamically Loading Spark Properties).
238
+
<br /><em>Note:</em> In client mode, this config must not be set through the <code>SparkConf</code>
239
+
directly in your application, because the driver JVM has already started at that point.
240
+
Instead, please set this through the default properties file.</td>
234
241
</td>
235
242
</tr>
236
243
<tr>
@@ -239,7 +246,9 @@ Apart from these, the following properties are also available, and may be useful
239
246
<td>
240
247
Set a special library path to use when launching the driver JVM.
241
248
242
-
<br /><em>Note:</em> setting this with <code>conf.set(...)</code> only works in <code>cluster</code> mode (e.g. YARN deployment). For <code>client</code> driver memory should be configured in <code>conf/spark-defaults.conf</code> or via the run-time settings (See Dynamically Loading Spark Properties).
249
+
<br /><em>Note:</em> In client mode, this config must not be set through the <code>SparkConf</code>
250
+
directly in your application, because the driver JVM has already started at that point.
251
+
Instead, please set this through the default properties file.</td>
0 commit comments