From 53cff5840923af81e048c0333117dc48163e4408 Mon Sep 17 00:00:00 2001
From: Tsuyoshi Ozawa <ozawa.tsuyoshi@gmail.com>
Date: Sat, 13 Dec 2014 16:34:25 +0900
Subject: [PATCH 1/2] Adding a documentation about dynamic resource allocation.

Signed-off-by: Tsuyoshi Ozawa <ozawa.tsuyoshi@gmail.com>
---
 docs/configuration.md | 49 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/docs/configuration.md b/docs/configuration.md
index 64aa94f622af..0e8eff405e71 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1142,6 +1142,55 @@ Apart from these, the following properties are also available, and may be useful
 </tr>
 </table>
 
+#### Dynamic allocation
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+<tr>
+  <td><code>spark.dynamicAllocation.enabled</code></td>
+  <td>false</td>
+  <td>
+    Enabling dynamic resource allocation.
+    Note that spark.shuffle.service.enabled need to be also true if cluster mode is YARN.
+  </td>
+</tr>
+<tr>
+  <td><code>spark.dynamicAllocation.minExecutors</code></td>
+  <td></td>
+  <td>
+    Minimum value of executors when dynamic allocation is enabled.
+  </td>
+</tr>
+<tr>
+  <td><code>spark.dynamicAllocation.maxExecutors</code></td>
+  <td></td>
+  <td>
+    Maximum value of executors when dynamic allocation is enabled.
+  </td>
+</tr>
+<tr>
+  <td><code>spark.dynamicAllocation.schedulerBacklogTimeout </code></td>
+  <td></td>
+  <td>
+    If there are backlogged tasks for this duration, add new executors.
+  </td>
+</tr>
+<tr>
+  <td><code>spark.dynamicAllocation.sustainedSchedulerBacklogTimeout</code></td>
+  <td></td>
+  <td>
+    If the backlog is sustained for this duration, add more executors
+    This is used only after the initial backlog timeout is exceeded.
+  </td>
+</tr>
+<tr>
+  <td><code>spark.dynamicAllocation.executorIdleTimeout</code></td>
+  <td></td>
+  <td>
+    If an executor has been idle for this duration, remove it.
+  </td>
+</tr>
+</table>
+
 #### Cluster Managers
 Each cluster manager in Spark has additional configuration options. Configurations
 can be found on the pages for each mode:

From 6827b56e37e25d64ce52af1caad742563c2bdb40 Mon Sep 17 00:00:00 2001
From: Tsuyoshi Ozawa <ozawa.tsuyoshi@gmail.com>
Date: Wed, 17 Dec 2014 02:14:15 +0900
Subject: [PATCH 2/2] Fixing a documentation of
 spark.dynamicAllocation.enabled.

Signed-off-by: Tsuyoshi Ozawa <ozawa.tsuyoshi@gmail.com>
---
 docs/configuration.md | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)

diff --git a/docs/configuration.md b/docs/configuration.md
index 0e8eff405e71..dbd1d095d07a 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1149,42 +1149,54 @@ Apart from these, the following properties are also available, and may be useful
   <td><code>spark.dynamicAllocation.enabled</code></td>
   <td>false</td>
   <td>
-    Enabling dynamic resource allocation.
+    Enabling dynamic allocations and removals of executors based on the workload.
+    The add policy depends on whether there are backlogged tasks waiting to be scheduled. If
+    the scheduler queue is not drained in N seconds, then new executors are added.
+    N is configured via spark.dynamicAllocation.schedulerBacklogTimeout and
+    spark.dynamicAllocation.sustainedSchedulerBacklogTimeout.
+    If the queue persists for another M seconds, then more executors are added and so on.
+    M is configured via spark.dynamicAllocation.executorIdleTimeout.
+    The number added in each round increases exponentially from the previous round until an upper bound on the
+    number of executors has been reached. The upper bound is based both on a configured property
+    and on the number of tasks pending: the policy will never increase the number of executor
+    requests past the number needed to handle all pending tasks.
+    To enable this feature, spark.dynamicAllocation.minExecutors and
+    spark.dynamicAllocation.maxExecutors must be configured.
     Note that spark.shuffle.service.enabled need to be also true if cluster mode is YARN.
   </td>
 </tr>
 <tr>
   <td><code>spark.dynamicAllocation.minExecutors</code></td>
-  <td></td>
+  <td>-1</td>
   <td>
     Minimum value of executors when dynamic allocation is enabled.
   </td>
 </tr>
 <tr>
   <td><code>spark.dynamicAllocation.maxExecutors</code></td>
-  <td></td>
+  <td>-1</td>
   <td>
     Maximum value of executors when dynamic allocation is enabled.
   </td>
 </tr>
 <tr>
-  <td><code>spark.dynamicAllocation.schedulerBacklogTimeout </code></td>
-  <td></td>
+  <td><code>spark.dynamicAllocation.schedulerBacklogTimeout</code></td>
+  <td>60</td>
   <td>
     If there are backlogged tasks for this duration, add new executors.
   </td>
 </tr>
 <tr>
   <td><code>spark.dynamicAllocation.sustainedSchedulerBacklogTimeout</code></td>
-  <td></td>
+  <td>60</td>
   <td>
-    If the backlog is sustained for this duration, add more executors
+    If the backlog is sustained for this duration, add more executors.
     This is used only after the initial backlog timeout is exceeded.
   </td>
 </tr>
 <tr>
   <td><code>spark.dynamicAllocation.executorIdleTimeout</code></td>
-  <td></td>
+  <td>600</td>
   <td>
     If an executor has been idle for this duration, remove it.
   </td>