Skip to content

[SPARK-4839][YARN] Adding a documentation about dynamic resource allocation. #3689

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

[SPARK-4839][YARN] Adding a documentation about dynamic resource allocation. #3689

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
53cff58
Adding a documentation about dynamic resource allocation.
oza Dec 13, 2014
6827b56
Fixing a documentation of spark.dynamicAllocation.enabled.
oza Dec 16, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1142,6 +1142,67 @@ Apart from these, the following properties are also available, and may be useful
</tr>
</table>

#### Dynamic allocation
<table class="table">
<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
<tr>
<td><code>spark.dynamicAllocation.enabled</code></td>
<td>false</td>
<td>
Enabling dynamic allocations and removals of executors based on the workload.
The add policy depends on whether there are backlogged tasks waiting to be scheduled. If
the scheduler queue is not drained in N seconds, then new executors are added.
N is configured via spark.dynamicAllocation.schedulerBacklogTimeout and
spark.dynamicAllocation.sustainedSchedulerBacklogTimeout.
If the queue persists for another M seconds, then more executors are added and so on.
M is configured via spark.dynamicAllocation.executorIdleTimeout.
The number added in each round increases exponentially from the previous round until an upper bound on the
number of executors has been reached. The upper bound is based both on a configured property
and on the number of tasks pending: the policy will never increase the number of executor
requests past the number needed to handle all pending tasks.
To enable this feature, spark.dynamicAllocation.minExecutors and
spark.dynamicAllocation.maxExecutors must be configured.
Note that spark.shuffle.service.enabled need to be also true if cluster mode is YARN.
</td>
</tr>
<tr>
<td><code>spark.dynamicAllocation.minExecutors</code></td>
<td>-1</td>
<td>
Minimum value of executors when dynamic allocation is enabled.
</td>
</tr>
<tr>
<td><code>spark.dynamicAllocation.maxExecutors</code></td>
<td>-1</td>
<td>
Maximum value of executors when dynamic allocation is enabled.
</td>
</tr>
<tr>
<td><code>spark.dynamicAllocation.schedulerBacklogTimeout</code></td>
<td>60</td>
<td>
If there are backlogged tasks for this duration, add new executors.
</td>
</tr>
<tr>
<td><code>spark.dynamicAllocation.sustainedSchedulerBacklogTimeout</code></td>
<td>60</td>
<td>
If the backlog is sustained for this duration, add more executors.
This is used only after the initial backlog timeout is exceeded.
</td>
</tr>
<tr>
<td><code>spark.dynamicAllocation.executorIdleTimeout</code></td>
<td>600</td>
<td>
If an executor has been idle for this duration, remove it.
</td>
</tr>
</table>

#### Cluster Managers
Each cluster manager in Spark has additional configuration options. Configurations
can be found on the pages for each mode:
Expand Down