Skip to content

Commit c1839c9

Browse files
tnachenAndrew Or
authored andcommitted
[SPARK-14645][MESOS] Fix python running on cluster mode mesos to have non local uris
## What changes were proposed in this pull request? Fix SparkSubmit to allow non-local python uris ## How was this patch tested? Manually tested with mesos-spark-dispatcher Author: Timothy Chen <[email protected]> Closes #12403 from tnachen/enable_remote_python.
1 parent a8d56f5 commit c1839c9

File tree

1 file changed

+3
-2
lines changed

1 file changed

+3
-2
lines changed

core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -305,8 +305,9 @@ object SparkSubmit {
305305
}
306306

307307
// Require all python files to be local, so we can add them to the PYTHONPATH
308-
// In YARN cluster mode, python files are distributed as regular files, which can be non-local
309-
if (args.isPython && !isYarnCluster) {
308+
// In YARN cluster mode, python files are distributed as regular files, which can be non-local.
309+
// In Mesos cluster mode, non-local python files are automatically downloaded by Mesos.
310+
if (args.isPython && !isYarnCluster && !isMesosCluster) {
310311
if (Utils.nonLocalPaths(args.primaryResource).nonEmpty) {
311312
printErrorAndExit(s"Only local python files are supported: $args.primaryResource")
312313
}

0 commit comments

Comments
 (0)