From fcfdb92428fc929a377d81e15dec2f4df832647d Mon Sep 17 00:00:00 2001 From: Kousuke Saruta Date: Sun, 10 May 2015 19:11:45 +0900 Subject: [PATCH 1/2] Updated doc for PySpark on YARN with cluster mode --- docs/submitting-applications.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/submitting-applications.md b/docs/submitting-applications.md index 3ecbf2308cd44..595bea53d7af8 100644 --- a/docs/submitting-applications.md +++ b/docs/submitting-applications.md @@ -59,7 +59,7 @@ for applications that involve the REPL (e.g. Spark shell). Alternatively, if your application is submitted from a machine far from the worker machines (e.g. locally on your laptop), it is common to use `cluster` mode to minimize network latency between the drivers and the executors. Note that `cluster` mode is currently not supported for -Mesos clusters or Python applications. +Mesos clusters or Python applications except for ones which run on YARN clusters. For Python applications, simply pass a `.py` file in the place of `` instead of a JAR, and add Python `.zip`, `.egg` or `.py` files to the search path with `--py-files`. From ad9f88cb62e2d9261bf630996736ae6ac6ac8188 Mon Sep 17 00:00:00 2001 From: Kousuke Saruta Date: Tue, 12 May 2015 03:32:16 +0900 Subject: [PATCH 2/2] Brushed up sentences --- docs/submitting-applications.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/submitting-applications.md b/docs/submitting-applications.md index 595bea53d7af8..e58645274e525 100644 --- a/docs/submitting-applications.md +++ b/docs/submitting-applications.md @@ -59,7 +59,7 @@ for applications that involve the REPL (e.g. Spark shell). Alternatively, if your application is submitted from a machine far from the worker machines (e.g. locally on your laptop), it is common to use `cluster` mode to minimize network latency between the drivers and the executors. Note that `cluster` mode is currently not supported for -Mesos clusters or Python applications except for ones which run on YARN clusters. +Mesos clusters. Currently only YARN supports cluster mode for Python applications. For Python applications, simply pass a `.py` file in the place of `` instead of a JAR, and add Python `.zip`, `.egg` or `.py` files to the search path with `--py-files`.