From cfe0f69676cc57922ed2076435a10e6f2355a540 Mon Sep 17 00:00:00 2001 From: frreiss Date: Wed, 7 Sep 2016 15:43:00 -0700 Subject: [PATCH 1/6] Documenting the current treatment of MAVEN_OPTS. --- docs/building-spark.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/building-spark.md b/docs/building-spark.md index 6908fc1ba74d..65b220e2336f 100644 --- a/docs/building-spark.md +++ b/docs/building-spark.md @@ -16,11 +16,11 @@ Building Spark using Maven requires Maven 3.3.9 or newer and Java 7+. ### Setting up Maven's Memory Usage -You'll need to configure Maven to use more memory than usual by setting `MAVEN_OPTS`. We recommend the following settings: +If you are compiling with Java 7, you'll need to configure Maven to use more memory than usual by setting `MAVEN_OPTS`: export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m" -If you don't run this, you may see errors like the following: +If you don't add these parameters to `MAVEN_OPTS`, you may see errors like the following: [INFO] Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-{{site.SCALA_BINARY_VERSION}}/classes... [ERROR] PermGen space -> [Help 1] @@ -28,12 +28,15 @@ If you don't run this, you may see errors like the following: [INFO] Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-{{site.SCALA_BINARY_VERSION}}/classes... [ERROR] Java heap space -> [Help 1] -You can fix this by setting the `MAVEN_OPTS` variable as discussed before. +You can fix these problems by setting the `MAVEN_OPTS` variable as discussed before. **Note:** -* For Java 8 and above this step is not required. -* If using `build/mvn` with no `MAVEN_OPTS` set, the script will automate this for you. +* This step is not needed for Java 8. +* If using `build/mvn` with no `MAVEN_OPTS` set, the script will automatically add the above options to the `MAVEN_OPTS` environment variable. +* The `test` phase of the Spark build will automatically add these options to `MAVEN_OPTS`, even when not using `build/mvn`. +* You may see warnings like "ignoring option MaxPermSize=1g; support was removed in 8.0" when building or running tests with Java 8. These warnings are harmless. + ### build/mvn From 25d4e993495df66b82de69ca1261523de2ce97a2 Mon Sep 17 00:00:00 2001 From: frreiss Date: Fri, 9 Sep 2016 13:13:32 -0700 Subject: [PATCH 2/6] Adding recommended memory options for Java 8 builds. --- docs/building-spark.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/building-spark.md b/docs/building-spark.md index 65b220e2336f..35807b9320f9 100644 --- a/docs/building-spark.md +++ b/docs/building-spark.md @@ -20,6 +20,10 @@ If you are compiling with Java 7, you'll need to configure Maven to use more mem export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m" +When compiling with Java 8, a similar but smaller set of memory parameters is necessary: + + export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m" + If you don't add these parameters to `MAVEN_OPTS`, you may see errors like the following: [INFO] Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-{{site.SCALA_BINARY_VERSION}}/classes... @@ -33,7 +37,7 @@ You can fix these problems by setting the `MAVEN_OPTS` variable as discussed bef **Note:** * This step is not needed for Java 8. -* If using `build/mvn` with no `MAVEN_OPTS` set, the script will automatically add the above options to the `MAVEN_OPTS` environment variable. +* If using `build/mvn` with no `MAVEN_OPTS` set, the script will automatically add the above options for Java 7 to the `MAVEN_OPTS` environment variable. * The `test` phase of the Spark build will automatically add these options to `MAVEN_OPTS`, even when not using `build/mvn`. * You may see warnings like "ignoring option MaxPermSize=1g; support was removed in 8.0" when building or running tests with Java 8. These warnings are harmless. From 4e2c92bee4c4063b0ba3119340e03a4d7ad1970c Mon Sep 17 00:00:00 2001 From: frreiss Date: Tue, 20 Sep 2016 23:08:34 -0700 Subject: [PATCH 3/6] Updated options after testing on multiple versions of Java. --- docs/building-spark.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/building-spark.md b/docs/building-spark.md index 35807b9320f9..ced5a4e1f239 100644 --- a/docs/building-spark.md +++ b/docs/building-spark.md @@ -24,7 +24,7 @@ When compiling with Java 8, a similar but smaller set of memory parameters is ne export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m" -If you don't add these parameters to `MAVEN_OPTS`, you may see errors like the following: +If you don't add these parameters to `MAVEN_OPTS`, you may see errors and warnings like the following: [INFO] Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-{{site.SCALA_BINARY_VERSION}}/classes... [ERROR] PermGen space -> [Help 1] @@ -32,11 +32,14 @@ If you don't add these parameters to `MAVEN_OPTS`, you may see errors like the f [INFO] Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-{{site.SCALA_BINARY_VERSION}}/classes... [ERROR] Java heap space -> [Help 1] + [INFO] Compiling 233 Scala sources and 41 Java sources to /Users/me/Development/spark/sql/core/target/scala-{site.SCALA_BINARY_VERSION}/classes... + OpenJDK 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled. + OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize= + You can fix these problems by setting the `MAVEN_OPTS` variable as discussed before. **Note:** -* This step is not needed for Java 8. * If using `build/mvn` with no `MAVEN_OPTS` set, the script will automatically add the above options for Java 7 to the `MAVEN_OPTS` environment variable. * The `test` phase of the Spark build will automatically add these options to `MAVEN_OPTS`, even when not using `build/mvn`. * You may see warnings like "ignoring option MaxPermSize=1g; support was removed in 8.0" when building or running tests with Java 8. These warnings are harmless. From 3dd1dba96dd0dd3c06ae140314eda3d865430a62 Mon Sep 17 00:00:00 2001 From: frreiss Date: Tue, 20 Sep 2016 23:15:32 -0700 Subject: [PATCH 4/6] Additional clarification about warning. --- docs/building-spark.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/building-spark.md b/docs/building-spark.md index ced5a4e1f239..c04e2efa7145 100644 --- a/docs/building-spark.md +++ b/docs/building-spark.md @@ -42,7 +42,7 @@ You can fix these problems by setting the `MAVEN_OPTS` variable as discussed bef * If using `build/mvn` with no `MAVEN_OPTS` set, the script will automatically add the above options for Java 7 to the `MAVEN_OPTS` environment variable. * The `test` phase of the Spark build will automatically add these options to `MAVEN_OPTS`, even when not using `build/mvn`. -* You may see warnings like "ignoring option MaxPermSize=1g; support was removed in 8.0" when building or running tests with Java 8. These warnings are harmless. +* You may see warnings like "ignoring option MaxPermSize=1g; support was removed in 8.0" when building or running tests with Java 8 and `build/mvn`. These warnings are harmless. ### build/mvn From 774a10108c9a1dcc19f7a9fe4ba0cdcc395aea8d Mon Sep 17 00:00:00 2001 From: frreiss Date: Tue, 20 Sep 2016 23:19:40 -0700 Subject: [PATCH 5/6] Shortened part about Java 7 opts. --- docs/building-spark.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/building-spark.md b/docs/building-spark.md index c04e2efa7145..f7c47547a033 100644 --- a/docs/building-spark.md +++ b/docs/building-spark.md @@ -16,14 +16,12 @@ Building Spark using Maven requires Maven 3.3.9 or newer and Java 7+. ### Setting up Maven's Memory Usage -If you are compiling with Java 7, you'll need to configure Maven to use more memory than usual by setting `MAVEN_OPTS`: - - export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m" - -When compiling with Java 8, a similar but smaller set of memory parameters is necessary: +You'll need to configure Maven to use more memory than usual by setting `MAVEN_OPTS`: export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m" +When compiling with Java 7, you will need to add the additional option "-XX:MaxPermSize=512M" to MAVEN_OPTS. + If you don't add these parameters to `MAVEN_OPTS`, you may see errors and warnings like the following: [INFO] Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-{{site.SCALA_BINARY_VERSION}}/classes... From 53a09cd5783d55048b2cf7579cf53ccc76bdf3d7 Mon Sep 17 00:00:00 2001 From: frreiss Date: Wed, 21 Sep 2016 11:35:22 -0700 Subject: [PATCH 6/6] Clarifying a bullet point. --- docs/building-spark.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/building-spark.md b/docs/building-spark.md index f7c47547a033..75c304a3ccec 100644 --- a/docs/building-spark.md +++ b/docs/building-spark.md @@ -38,7 +38,7 @@ You can fix these problems by setting the `MAVEN_OPTS` variable as discussed bef **Note:** -* If using `build/mvn` with no `MAVEN_OPTS` set, the script will automatically add the above options for Java 7 to the `MAVEN_OPTS` environment variable. +* If using `build/mvn` with no `MAVEN_OPTS` set, the script will automatically add the above options to the `MAVEN_OPTS` environment variable. * The `test` phase of the Spark build will automatically add these options to `MAVEN_OPTS`, even when not using `build/mvn`. * You may see warnings like "ignoring option MaxPermSize=1g; support was removed in 8.0" when building or running tests with Java 8 and `build/mvn`. These warnings are harmless.