SPARK-2641: Passing num executors to spark arguments from properties file

Since we can set spark executor memory and executor cores using property file, we must also be allowed to set the executor instances.

Author: Kanwaljit Singh <[email protected]>

Closes #1657 from kjsingh/branch-1.0 and squashes the following commits:

d8a5a12 [Kanwaljit Singh] SPARK-2641: Fixing how spark arguments are loaded from properties file for num executors

Author: Kanwaljit Singh

core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala

@@ -105,6 +105,8 @@ private[spark] class SparkSubmitArguments(args: Seq[String]) {
       .getOrElse(defaultProperties.get("spark.cores.max").orNull)
     name = Option(name).getOrElse(defaultProperties.get("spark.app.name").orNull)
     jars = Option(jars).getOrElse(defaultProperties.get("spark.jars").orNull)
+    numExecutors = Option(numExecutors)
+      .getOrElse(defaultProperties.get("spark.executor.instances").orNull)
 
     // This supports env vars in older versions of Spark
     master = Option(master).getOrElse(System.getenv("MASTER"))