Skip to content

Commit d850b4b

Browse files
jerryshaoAndrew Or
authored andcommitted
[SPARK-6975][Yarn] Fix argument validation error
`numExecutors` checking is failed when dynamic allocation is enabled with default configuration. Details can be seen is [SPARK-6975](https://issues.apache.org/jira/browse/SPARK-6975). sryza, please help me to review this, not sure is this the correct way, I think previous you change this part :) Author: jerryshao <[email protected]> Closes #5551 from jerryshao/SPARK-6975 and squashes the following commits: 4335da1 [jerryshao] Change according to the comments 77bdcbd [jerryshao] Fix argument validation error
1 parent 1991337 commit d850b4b

File tree

1 file changed

+6
-2
lines changed

1 file changed

+6
-2
lines changed

yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -103,9 +103,13 @@ private[spark] class ClientArguments(args: Array[String], sparkConf: SparkConf)
103103
* This is intended to be called only after the provided arguments have been parsed.
104104
*/
105105
private def validateArgs(): Unit = {
106-
if (numExecutors <= 0) {
106+
if (numExecutors < 0 || (!isDynamicAllocationEnabled && numExecutors == 0)) {
107107
throw new IllegalArgumentException(
108-
"You must specify at least 1 executor!\n" + getUsageMessage())
108+
s"""
109+
|Number of executors was $numExecutors, but must be at least 1
110+
|(or 0 if dynamic executor allocation is enabled).
111+
|${getUsageMessage()}
112+
""".stripMargin)
109113
}
110114
if (executorCores < sparkConf.getInt("spark.task.cpus", 1)) {
111115
throw new SparkException("Executor cores must not be less than " +

0 commit comments

Comments
 (0)