[SPARKR][SPARK-10711] Do not assume spark.submit.deployMode is always set

[falaki]

In RUtils.sparkRPackagePath() we

1. Call sys.props("spark.submit.deployMode") which returns null if spark.submit.deployMode is not suet
2. Call sparkConf.get("spark.submit.deployMode") which throws NoSuchElementException if spark.submit.deployMode is not set. This patch simply passes a default value ("cluster") for spark.submit.deployMode.

cc @rxin

[falaki]

cc @shivaram

[shivaram]

Any reason this would not be set ? My assumption was that all spark-submit applications had this -- so I guess this is for applications not using spark-submit ?

cc @sun-rui who added the function

[falaki]

Yes, if an application starts the JVM manually (not using spark-submit) this call will fail. The change just adds safety without compromising functionality or correctness.

[SparkQA]

Test build #42699 has finished for PR 8832 at commit 8b80886.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

if anything the default deploy mode should be client to be consistent with assumptions in the code elsewhere

[sun-rui]

@falaki , are you using RStudio, where spark-submit is not involved?

I don't think we can simply set a default value. Because it is possible that the default mode does not match spark master. For example, if the spark master is 'yarn-cluster', the default mode is "client", that does not match.

The correct policy is:
if the master does not contain deploy mode information, the deploy mode is "client",
else get deploy mode from the master URL.

I think this issue is not related to SparkR only. @andrewor14, any policy when handling default configurations when a spark-application is not launched via spark-submit? Should we extract some logic from spark-submit so that the logic can be called in non-spark-submit cases to keep consistency?

[andrewor14]

retest this please

[andrewor14]

if the master does not contain deploy mode information, the deploy mode is "client",
else get deploy mode from the master URL.

We're actually deprecating the master URLs yarn-client and yarn-cluster (#8385). In general I find it pretty confusing to have the deploy mode embedded in the master URL.

I think this issue is not related to SparkR only. @andrewor14, any policy when handling default configurations when a spark-application is not launched via spark-submit? Should we extract some logic from spark-submit so that the logic can be called in non-spark-submit cases to keep consistency?

We do have the launcher library, but beyond that I don't think this is something we support because it gets pretty difficult to maintain. AFAIK deploy mode is the only such config so this should be fine.

[andrewor14]

LGTM merging once tests pass

[SparkQA]

Test build #42800 has finished for PR 8832 at commit a643281.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

I've merged this.