From b7d446f9588fead34cb96230afded3a847a29cba Mon Sep 17 00:00:00 2001 From: Andrew Or Date: Wed, 7 May 2014 11:43:42 -0700 Subject: [PATCH 1/2] Warn about PySpark on YARN on Red Hat --- docs/building-with-maven.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/building-with-maven.md b/docs/building-with-maven.md index cac01ded60d9..2063f063244d 100644 --- a/docs/building-with-maven.md +++ b/docs/building-with-maven.md @@ -98,7 +98,6 @@ The ScalaTest plugin also supports running only a specific test suite as follows $ mvn -Dhadoop.version=... -Dsuites=org.apache.spark.repl.ReplSuite test - ## Continuous Compilation ## We use the scala-maven-plugin which supports incremental and continuous compilation. E.g. @@ -129,6 +128,10 @@ Java 8 tests are run when -Pjava8-tests profile is enabled, they will run in spi For these tests to run your system must have a JDK 8 installation. If you have JDK 8 installed but it is not the system default, you can set JAVA_HOME to point to JDK 8 before running the tests. +## Building for PySpark on YARN ## + +There is a known problem with building an assembly jar for running PySpark on YARN on Red Hat based operating systems. If you wish to run PySpark on a YARN cluster with Red Hat installed, we recommend that you build the jar elsewhere, then ship it over to the cluster. We are investigating the exact cause for this. + ## Packaging without Hadoop dependencies for deployment on YARN ## The assembly jar produced by "mvn package" will, by default, include all of Spark's dependencies, including Hadoop and some of its ecosystem projects. On YARN deployments, this causes multiple versions of these to appear on executor classpaths: the version packaged in the Spark assembly and the version on each node, included with yarn.application.classpath. The "hadoop-provided" profile builds the assembly without including Hadoop-ecosystem projects, like ZooKeeper and Hadoop itself. From 21c24927bec38885395616eb7cb351ba2358ac24 Mon Sep 17 00:00:00 2001 From: Andrew Or Date: Wed, 7 May 2014 22:08:04 -0700 Subject: [PATCH 2/2] Add JIRA reference + a note about requiring maven --- docs/building-with-maven.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/building-with-maven.md b/docs/building-with-maven.md index 2063f063244d..3a06f419f743 100644 --- a/docs/building-with-maven.md +++ b/docs/building-with-maven.md @@ -130,7 +130,7 @@ If you have JDK 8 installed but it is not the system default, you can set JAVA_H ## Building for PySpark on YARN ## -There is a known problem with building an assembly jar for running PySpark on YARN on Red Hat based operating systems. If you wish to run PySpark on a YARN cluster with Red Hat installed, we recommend that you build the jar elsewhere, then ship it over to the cluster. We are investigating the exact cause for this. +PySpark on YARN is only supported if the jar is built with maven. Further, there is a known problem with building this assembly jar on Red Hat based operating systems (see SPARK-1753). If you wish to run PySpark on a YARN cluster with Red Hat installed, we recommend that you build the jar elsewhere, then ship it over to the cluster. We are investigating the exact cause for this. ## Packaging without Hadoop dependencies for deployment on YARN ##