SPARK-1652: Set driver memory correctly in spark-submit.

[pwendell]

The previous check didn't account for the fact that the default
deploy mode is "client" unless otherwise specified. Also, this
sets the more narrowly defined SPARK_DRIVER_MEMORY instead of setting
SPARK_MEM.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[pwendell]

This was reported by @witgo.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14884/

[aarondav]

I'm actually a little nervous about the default value of DEPLOY_MODE. Do you think we could put an explicit line in the shell script like

# Default deploy mode is "client".
DEPLOY_MODE=${DEPLOY_MODE:-"client"}

We can keep the rest of your solution as-is, I just like having a 2-valued DEPLOY_MODE better than a 3-valued one.

[pwendell]

@aarondav - would never want to make you nervous. I made the suggested change. Mind taking a look?

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14891/

[aarondav]

Sorry to not notice this earlier,but given the default value, I think the equality check with "client" now may make more sense.

Also, along similar lines, perhaps we could use -n instead of ! -z for DRIVER_MEMORY, and also wrap the variables in quotes. For instance, right now if $DRIVER_MEMORY is not set, this is actually evaluating if [ ! -z ], which happens to evaluate to false, but is confusing because if [ ! -n ] also evaluates to false. By putting quotes around it, we'll make sure we actually evaluate if [ -n "" ] if it's empty, which has the semantics we're actually looking for.

[pwendell]

@aarondav check again? Responded to your feedback.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14892/

[aarondav]

LGTM, merged into master and branch-1.0. Thanks!!