Document the configs as well

Author: Andrew Or

docs/configuration.md

@@ -1008,6 +1008,67 @@ Apart from these, the following properties are also available, and may be useful
 </tr>
 </table>
 
+#### Dynamic allocation
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+<tr>
+  <td><code>spark.dynamicAllocation.enabled</code></td>
+  <td>false</td>
+  <td>
+    Whether to use dynamic resource allocation, which scales the number of executors registered
+    with this application up and down based on the workload. Note that this is currently only
+    available on YARN mode. For more detail, see the description
+    <a href="job-scheduling.html#dynamic-resource-allocation">here</a>.
+    <br><br>
+    This requires the following configurations to be set:
+    <code>spark.dynamicAllocation.minExecutors</code>,
+    <code>spark.dynamicAllocation.maxExecutors</code>, and
+    <code>spark.shuffle.service.enabled</code>
+  </td>
+</tr>
+<tr>
+  <td><code>spark.dynamicAllocation.minExecutors</code></td>
+  <td>(none)</td>
+  <td>
+    Lower bound for the number of executors if dynamic allocation is enabled (required).
+  </td>
+</tr>
+<tr>
+  <td><code>spark.dynamicAllocation.maxExecutors</code></td>
+  <td>(none)</td>
+  <td>
+    Upper bound for the number of executors if dynamic allocation is enabled (required).
+  </td>
+</tr>
+<tr>
+  <td><code>spark.dynamicAllocation.schedulerBacklogTimeout</code></td>
+  <td>60</td>
+  <td>
+    If dynamic allocation is enabled and there have been pending tasks backlogged for more than
+    this duration (in seconds), new executors will be requested. For more detail, see this
+    <a href="job-scheduling.html#resource-allocation-policy">description</a>.
+  </td>
+</tr>
+<tr>
+  <td><code>spark.dynamicAllocation.sustainedSchedulerBacklogTimeout</code></td>
+  <td><code>schedulerBacklogTimeout</code></td>
+  <td>
+    Same as <code>spark.dynamicAllocation.schedulerBacklogTimeout</code>, but used only for
+    subsequent executor requests. For more detail, see this
+    <a href="job-scheduling.html#resource-allocation-policy">description</a>.
+  </td>
+</tr>
+<tr>
+  <td><code>spark.dynamicAllocation.executorIdleTimeout</code></td>
+  <td>600</td>
+  <td>
+    If dynamic allocation is enabled and an executor has been idle for more than this duration
+    (in seconds), the executor will be removed. For more detail, see this
+    <a href="job-scheduling.html#resource-allocation-policy">description</a>.
+  </td>
+</tr>
+</table>
+
 #### Security
 <table class="table">
 <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
@@ -1142,67 +1203,6 @@ Apart from these, the following properties are also available, and may be useful
 </tr>
 </table>
 
-#### Dynamic allocation
-<table class="table">
-<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
-<tr>
-  <td><code>spark.dynamicAllocation.enabled</code></td>
-  <td>false</td>
-  <td>
-    Enabling dynamic allocations and removals of executors based on the workload.
-    The add policy depends on whether there are backlogged tasks waiting to be scheduled. If
-    the scheduler queue is not drained in N seconds, then new executors are added.
-    N is configured via spark.dynamicAllocation.schedulerBacklogTimeout and
-    spark.dynamicAllocation.sustainedSchedulerBacklogTimeout.
-    If the queue persists for another M seconds, then more executors are added and so on.
-    M is configured via spark.dynamicAllocation.executorIdleTimeout.
-    The number added in each round increases exponentially from the previous round until an upper bound on the
-    number of executors has been reached. The upper bound is based both on a configured property
-    and on the number of tasks pending: the policy will never increase the number of executor
-    requests past the number needed to handle all pending tasks.
-    To enable this feature, spark.dynamicAllocation.minExecutors and
-    spark.dynamicAllocation.maxExecutors must be configured.
-    Note that spark.shuffle.service.enabled need to be also true if cluster mode is YARN.
-  </td>
-</tr>
-<tr>
-  <td><code>spark.dynamicAllocation.minExecutors</code></td>
-  <td>-1</td>
-  <td>
-    Minimum value of executors when dynamic allocation is enabled.
-  </td>
-</tr>
-<tr>
-  <td><code>spark.dynamicAllocation.maxExecutors</code></td>
-  <td>-1</td>
-  <td>
-    Maximum value of executors when dynamic allocation is enabled.
-  </td>
-</tr>
-<tr>
-  <td><code>spark.dynamicAllocation.schedulerBacklogTimeout</code></td>
-  <td>60</td>
-  <td>
-    If there are backlogged tasks for this duration, add new executors.
-  </td>
-</tr>
-<tr>
-  <td><code>spark.dynamicAllocation.sustainedSchedulerBacklogTimeout</code></td>
-  <td>60</td>
-  <td>
-    If the backlog is sustained for this duration, add more executors.
-    This is used only after the initial backlog timeout is exceeded.
-  </td>
-</tr>
-<tr>
-  <td><code>spark.dynamicAllocation.executorIdleTimeout</code></td>
-  <td>600</td>
-  <td>
-    If an executor has been idle for this duration, remove it.
-  </td>
-</tr>
-</table>
-
 #### Cluster Managers
 Each cluster manager in Spark has additional configuration options. Configurations
 can be found on the pages for each mode:

docs/job-scheduling.md

@@ -86,8 +86,8 @@ All configurations used by this feature live under the `spark.dynamicAllocation.
 To enable this feature, your application must set `spark.dynamicAllocation.enabled` to `true` and
 provide lower and upper bounds for the number of executors through
 `spark.dynamicAllocation.minExecutors` and `spark.dynamicAllocation.maxExecutors`. Other relevant
-configurations are described on the [configurations page](configuration.html) and in the subsequent
-sections in detail.
+configurations are described on the [configurations page](configuration.html#dynamic-allocation)
+and in the subsequent sections in detail.
 
 Additionally, your application must use an external shuffle service (described below). To enable
 this, set `spark.shuffle.service.enabled` to `true`. In YARN, this external shuffle service is