Skip to content

[SPARK-17421] [DOCS] Documenting the current treatment of MAVEN_OPTS. #15005

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

[SPARK-17421] [DOCS] Documenting the current treatment of MAVEN_OPTS. #15005

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
cfe0f69
Documenting the current treatment of MAVEN_OPTS.
frreiss Sep 7, 2016
25d4e99
Adding recommended memory options for Java 8 builds.
frreiss Sep 9, 2016
4e2c92b
Updated options after testing on multiple versions of Java.
frreiss Sep 21, 2016
3dd1dba
Additional clarification about warning.
frreiss Sep 21, 2016
774a101
Shortened part about Java 7 opts.
frreiss Sep 21, 2016
53a09cd
Clarifying a bullet point.
frreiss Sep 21, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 14 additions & 6 deletions docs/building-spark.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,24 +16,32 @@ Building Spark using Maven requires Maven 3.3.9 or newer and Java 7+.

### Setting up Maven's Memory Usage

You'll need to configure Maven to use more memory than usual by setting `MAVEN_OPTS`. We recommend the following settings:
You'll need to configure Maven to use more memory than usual by setting `MAVEN_OPTS`:

export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m"

If you don't run this, you may see errors like the following:
When compiling with Java 7, you will need to add the additional option "-XX:MaxPermSize=512M" to MAVEN_OPTS.

If you don't add these parameters to `MAVEN_OPTS`, you may see errors and warnings like the following:

[INFO] Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-{{site.SCALA_BINARY_VERSION}}/classes...
[ERROR] PermGen space -> [Help 1]

[INFO] Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-{{site.SCALA_BINARY_VERSION}}/classes...
[ERROR] Java heap space -> [Help 1]

You can fix this by setting the `MAVEN_OPTS` variable as discussed before.
[INFO] Compiling 233 Scala sources and 41 Java sources to /Users/me/Development/spark/sql/core/target/scala-{site.SCALA_BINARY_VERSION}/classes...
OpenJDK 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=

You can fix these problems by setting the `MAVEN_OPTS` variable as discussed before.
Copy link
Member

@srowen srowen Sep 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can go too. This is redundant if the structure is improved (over what it was already before your change, not because of your change)


**Note:**

* For Java 8 and above this step is not required.
* If using `build/mvn` with no `MAVEN_OPTS` set, the script will automate this for you.
* If using `build/mvn` with no `MAVEN_OPTS` set, the script will automatically add the above options to the `MAVEN_OPTS` environment variable.
* The `test` phase of the Spark build will automatically add these options to `MAVEN_OPTS`, even when not using `build/mvn`.
* You may see warnings like "ignoring option MaxPermSize=1g; support was removed in 8.0" when building or running tests with Java 8 and `build/mvn`. These warnings are harmless.


### build/mvn

Expand Down