diff --git a/docs/configuration.md b/docs/configuration.md index 64aa94f622af..dbd1d095d07a 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1142,6 +1142,67 @@ Apart from these, the following properties are also available, and may be useful +#### Dynamic allocation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Property NameDefaultMeaning
spark.dynamicAllocation.enabledfalse + Enabling dynamic allocations and removals of executors based on the workload. + The add policy depends on whether there are backlogged tasks waiting to be scheduled. If + the scheduler queue is not drained in N seconds, then new executors are added. + N is configured via spark.dynamicAllocation.schedulerBacklogTimeout and + spark.dynamicAllocation.sustainedSchedulerBacklogTimeout. + If the queue persists for another M seconds, then more executors are added and so on. + M is configured via spark.dynamicAllocation.executorIdleTimeout. + The number added in each round increases exponentially from the previous round until an upper bound on the + number of executors has been reached. The upper bound is based both on a configured property + and on the number of tasks pending: the policy will never increase the number of executor + requests past the number needed to handle all pending tasks. + To enable this feature, spark.dynamicAllocation.minExecutors and + spark.dynamicAllocation.maxExecutors must be configured. + Note that spark.shuffle.service.enabled need to be also true if cluster mode is YARN. +
spark.dynamicAllocation.minExecutors-1 + Minimum value of executors when dynamic allocation is enabled. +
spark.dynamicAllocation.maxExecutors-1 + Maximum value of executors when dynamic allocation is enabled. +
spark.dynamicAllocation.schedulerBacklogTimeout60 + If there are backlogged tasks for this duration, add new executors. +
spark.dynamicAllocation.sustainedSchedulerBacklogTimeout60 + If the backlog is sustained for this duration, add more executors. + This is used only after the initial backlog timeout is exceeded. +
spark.dynamicAllocation.executorIdleTimeout600 + If an executor has been idle for this duration, remove it. +
+ #### Cluster Managers Each cluster manager in Spark has additional configuration options. Configurations can be found on the pages for each mode: