SPARK-1691: Support quoted arguments inside of spark-submit.

[pwendell]

This is a fairly straightforward fix. The bug was reported by @vanzin and the fix was proposed by @deanwampler and myself. Please take a look!

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14603/

[pwendell]

Since this was already vetted by a few folks on the mailing list I'm gonna go ahead an merge it.

[cosmicangler]

Is this fix in 1.0 as I am still getting this error when submitting with yarn?

[vanzin]

@cosmicangler yes:
dd601bf

If you're running into some issue with submitting jobs, probably better to ask on the list or file a bug.

[cosmicangler]

Thank you for hte I can file a bug but my problem is this bug that marked fixed but not corrected in the 1.0 distribution I am using. I am unable to use quoted arguments for --driver-java-options as discussed in this bug.

Should a submit another identical bug to this one?

[vanzin]

Being the person who reported the original bug :-), the issue for me was not quotes inside --driver-java-options, but providing several arguments (e.g. --driver-java-options "-Dfoo -Dbar" would not work, as the example in the jira issue shows). So maybe the commit description is a little misleading.

That issue is definitely fixed, so it sounds like your problem is different (e.g. sounds like you want quotes to be propagated to the driver but that's not happening).

[cosmicangler]

That is correct. That is precisely what is not working for me int he 1.0 distribution. I am looking to do a --driver-java-options "-XX:+UseParallelOldGC -XX:parallelGCThreads=8"

I get -XX:parallelGCThreads=8 is not a class error as it doesn't appear to be respecting the " ". Maybe there is an issue the version our ops folks used in our clusters. Was this fixed in 1.0 master?

[vanzin]

That is working fine for me. Looks like an issue with your deployment. The fix is both in branch-1.0 and master (there's no such thing as "1.0 master").

[cosmicangler]

Thanks

[koertkuipers]

on the command line i can get this to work now, but its still way beyond my bash skills to use "exec spark-submit" inside a script with multiple java options. this is a fairly normal use case i imagine if you want to launch an application using spark-submit.

for example my attempt to launch SparkPi with multiple java options is below. when i run it i get:
$ ./pi
spark-submit --class org.apache.spark.examples.SparkPi --driver-java-options "-Dfoo=a -Dbar=b" /usr/lib/spark/examples/lib/spark-examples_2.10-1.0.0-cdh5.1.0.jar
Error: Could not find or load main class "-Dfoo=a

i have no doubt its me that is doing something wrong here, but it shouldnt be this hard to use... perhaps a different syntax would make life easier?
best, koert

$ more pi
#!/bin/bash -e

DRIVER_JAVA_OPTS="-Dfoo=a -Dbar=b"
SPARK_SUBMIT="spark-submit"

OPTS=""
OPTS+=" --class org.apache.spark.examples.SparkPi"
OPTS+=" --driver-java-options "${DRIVER_JAVA_OPTS}""

echo $SPARK_SUBMIT $OPTS /usr/lib/spark/examples/lib/spark-examples_2.10-1.0.0-cdh5.1.0.jar
exec $SPARK_SUBMIT $OPTS /usr/lib/spark/examples/lib/spark-examples_2.10-1.0.0-cdh5.1.0.jar

[ganeshm25]

@koertkuipers i am trying to do achieve running the multiple driver-java-options with Spark 1.4.2 inside a bash script? is there a solution you found for this ?

[koertkuipers]

@ganeshm25 it seems to work in newer spark versions. i havent tried in spark 1.4.2. however its still very tricky to get it right and i would prefer a simpler solution.

[ganeshm25]

@koertkuipers can you the most recent syntax that you used to achieve the same?

[koertkuipers]

OPTS+=" --driver-java-options \"-Da=b -Dx=y\""