Skip to content

Commit f508146

Browse files
committed
SPARK-1652: Set driver memory correctly in spark-submit.
The previous check didn't account for the fact that the default deploy mode is "client" unless otherwise specified. Also, this sets the more narrowly defined SPARK_DRIVER_MEM instead of setting SPARK_MEM.
1 parent 70bcdef commit f508146

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

bin/spark-submit

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,8 @@ while (($#)); do
3535
shift
3636
done
3737

38-
if [ ! -z $DRIVER_MEMORY ] && [ ! -z $DEPLOY_MODE ] && [ $DEPLOY_MODE = "client" ]; then
39-
export SPARK_MEM=$DRIVER_MEMORY
38+
if [ ! -z $DRIVER_MEMORY ] && [ ! "$DEPLOY_MODE" == "cluster" ]; then
39+
export SPARK_DRIVER_MEMORY=$DRIVER_MEMORY
4040
fi
4141

4242
$SPARK_HOME/bin/spark-class org.apache.spark.deploy.SparkSubmit "${ORIG_ARGS[@]}"

0 commit comments

Comments
 (0)