SPARK-2310. Support arbitrary Spark properties on the command line with ...

[sryza]

...spark-submit

The PR allows invocations like
spark-submit --class org.MyClass --spark.shuffle.spill false myjar.jar

[sryza]

Verified this on a pseudo-distributed cluster

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16222/

[pwendell]

Hey @sryza - I did a straw poll offline discussing this with a few other contributors. The consensus was that it might be better to have a --conf flag with an = sign instead of representing spark conf properties directly as flags.

I.e. --conf spark.app.name=blah

On admittedly bad thing about this approach is that if users have arguments with spaces in them, they will have to quote the entire thing:

./bin/spark-submit --conf "spark.app.name=My app"

Which might not be intuitive, so it would be good to document that (provided you are generally okay with this proposed syntax).

[lianhuiwang]

how about -Dspark.app.name=blah? because in jvm or Hadoop, they use -D flag to represent conf properties.

[srowen]

-D feels more natural indeed; I would expect those args to be passed through to the JVM as-is. Because that's a way to set these env properties too right? In fact, wouldn't it be nicer to just let any -D arg through?

[pwendell]

IMO -D does not have the right semantics here because the user isn't logically setting java properties for the submission tool, they are setting spark configuration properties for their application. The application might run totally remotely, for instance, so why should the user expect that a -D set on the submission site gets packaged up and sent to the remote launcher. Also confusing is that we'd only really triage -D options that are spark properties, not other ones, so the semantics would differ depending on whether the user happened to set a java property that started with spark. For these reasons I feel it's better to just have an explicit config-related flag.

[mateiz]

I agree, -D is for JVM options, but these are not arbitrary JVM options.

[srowen]

Good points. I meant triaging all -D options but yes those then have very 'local' semantics.

[sryza]

Updated patch to use --conf

[pwendell]

So here, it might be a bit nicer to do something like:

value match {
  case k :: "=" :: v =>
    sparkProperties(k) = v
  case _ =>
    // throw exception saying there is a bad format
}

Would this work?

[sryza]

That didn't compile for me. As far as I can tell you can't do pattern matching on strings in that way. Switching it to something that looks a little prettier. Let me know if I'm missing anything.

[sryza]

Updated patch addresses Patrick's feedback

[SparkQA]

QA tests have started for PR 1253. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16934/consoleFull

[SparkQA]

QA results for PR 1253:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16934/consoleFull

[pwendell]

LGTM - @mateiz @rxin any final comments here?

[rxin]

I think you want a val here?

[mateiz]

Can you support -c in addition to --conf?

Also, the spark-submit doc (http://spark.apache.org/docs/latest/submitting-applications.html) should be updated to list this option.

[SparkQA]

QA tests have started for PR 1253. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16971/consoleFull

[SparkQA]

QA results for PR 1253:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16971/consoleFull

[sryza]

The failure appears to be unrelated (MIMA compatibility issue in MLLib).

[pwendell]

Jenkins, retest this please.

[SparkQA]

QA tests have started for PR 1253. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16988/consoleFull

[SparkQA]

QA results for PR 1253:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16988/consoleFull

[pwendell]

@sryza I merged this just now because another patch was going to change this code and I wanted to avoid you having to rebase again. That said, I found an issue with this after merging. Would you be able to fix this?

https://issues.apache.org/jira/browse/SPARK-2664

[sryza]

Thanks, I'm on it.