Match YARN behavior for spark.jars and spark.files

[ash211]

Background:

• spark.jars has synonyms --jars in spark-submit.sh and addJar() on SparkLauncher
• same setup for files with: spark.files and --files and addFile()

Current behavior in YARN:

• spark.jars treats file:// URIs as local to the submitter, and local:// as local to the executor
• same with spark.files
• see https://spark.apache.org/docs/latest/submitting-applications.html#advanced-dependency-management

Current behavior in Kubernetes:

• spark.jars treats file:// URIs as local to the container. local:// has no meaning
• same with spark.files
• there's an additional parameter --upload-jars and spark.kubernetes.driver.uploads.jars that is used with URIs that are local to the submitter
• see https://github.com/apache-spark-on-k8s/spark/blob/k8s-support-alternate-incremental/docs/running-on-kubernetes.md#adding-other-jars

Desired changes in Kubernetes behavior:

• merge these two configs (spark.files and spark.kubernetes.driver.uploads.jars) into one for parallelism with YARN
• change the behavior of spark.jars so that it treats file:// as submitter-local and local:// as container-local
• remove --upload-jars and spark.kubernetes.driver.uploads.jars entirely and offer spark.jars=file:// as the replacement

Revealed when migrating an internal application to support both YARN and K8s and realized addFile() and addJar() don't work the same way between YARN and K8s.

[mccheah]

Should unspecified schemes be treated as local to the submitter or local to the container?

[ash211]

I believe in YARN spark.jars with no protocol, e.g. /path/to/file.jar, are treated as having the file:// protocol, so are local to the submitter.

I think that's what the docs mean by "absolute paths" in

file: - Absolute paths and file:/ URIs are served by the driver’s HTTP file server, and every executor pulls the file from the driver HTTP server.

So yes, I think unspecified schemas in spark.jars and spark.files should be treated as local to the submitter. They probably did this in YARN for bash autocomplete locally

[mccheah]

Think this is a blocker for the alpha release if only because it's a major API decision.

[ash211]

Fixed in #107