Organize configuration docs #880

pwendell · 2014-05-26T05:30:52Z

This PR improves and organizes the config option page
and makes a few other changes to config docs. See a preview here:
http://people.apache.org/~pwendell/config-improvements/configuration.html

The biggest changes are:

The configs for the standalone master/workers were moved to the
standalone page and out of the general config doc.
SPARK_LOCAL_DIRS was missing from the standalone docs.
Expanded discussion of injecting configs with spark-submit, including an
example.
Config options were organized into the following categories:

Runtime Environment
Shuffle Behavior
Spark UI
Compression and Serialization
Execution Behavior
Networking
Scheduling
Security
Spark Streaming

This uses the following categories: - Runtime Environment - Shuffle Behavior - Spark UI - Compression and Serialization - Execution Behavior - Networking - Scheduling - Security - Spark Streaming

AmplabJenkins · 2014-05-26T05:32:58Z

Build triggered.

AmplabJenkins · 2014-05-26T05:33:06Z

Build started.

AmplabJenkins · 2014-05-26T05:37:58Z

Build triggered.

AmplabJenkins · 2014-05-26T05:38:06Z

Build started.

ash211 · 2014-05-26T05:45:49Z

Hi Patrick I just added some documentation for ports that Spark uses in the below pull request that was merged this afternoon. Can you make sure that info doesn't get lost in this update?

Much appreciation for the docs updates!

#856

pwendell · 2014-05-26T05:54:27Z

Hey Andrew, I actually meant to rebase on master before starting this precisely because of #856, but now I see that I missed it. I'll go back and make sure I include all your changes.

pwendell · 2014-05-26T06:01:25Z

@ash211 Hey by the way, are all of the changes in your PR just formatting related other than the port related section and the table of contents? I noticed you changed the linebreaks around in a bunch of places, was this just for line width or did you modify content as well?

AmplabJenkins · 2014-05-26T06:13:15Z

Build finished. All automated tests passed.

AmplabJenkins · 2014-05-26T06:13:15Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15201/

AmplabJenkins · 2014-05-26T06:17:58Z

Merged build triggered.

Conflicts: docs/configuration.md

AmplabJenkins · 2014-05-26T06:18:06Z

Merged build started.

AmplabJenkins · 2014-05-26T06:18:32Z

Build finished. All automated tests passed.

AmplabJenkins · 2014-05-26T06:18:32Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15202/

pwendell · 2014-05-26T06:21:13Z

@ash211 okay I think this is up-to-date with your changes. I did move your security stuff over to the standalone cluster page, since almost every entry was only relevant to standalone mode. I also might have dropped a few cases where you improved some in-code indentation... I'll try to find every last one but the diff was very complicated.

AmplabJenkins · 2014-05-26T06:22:58Z

Merged build triggered.

AmplabJenkins · 2014-05-26T06:23:06Z

Merged build started.

AmplabJenkins · 2014-05-26T06:28:12Z

Merged build finished.

AmplabJenkins · 2014-05-26T06:28:13Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15203/

ash211 · 2014-05-28T04:37:12Z

docs/configuration.md

Why bother having a configuration option if you can't change it?

That's a good question :P I think this used to be change-able but it was reverted. I'm not totally sure about the genealogy of this so I'm going to leave it, but it's worth seeing if we should just remove it.

ash211 · 2014-05-28T04:46:38Z

Thanks for the further docs organization @pwendell !

AmplabJenkins · 2014-05-28T22:02:58Z

Merged build triggered.

AmplabJenkins · 2014-05-28T22:03:04Z

Merged build started.

mateiz · 2014-05-28T22:09:50Z

docs/configuration.md

Should say YARN

mateiz · 2014-05-28T22:14:05Z

Made a few other small comments but it looks good otherwise, feel free to merge it when they're fixed.

AmplabJenkins · 2014-05-28T22:43:21Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-28T22:43:21Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15266/

AmplabJenkins · 2014-05-28T22:47:58Z

Merged build triggered.

AmplabJenkins · 2014-05-28T22:48:04Z

Merged build started.

pwendell · 2014-05-28T22:49:38Z

Okay I'm going to merge this. Thanks Matei and Andrew!

This PR improves and organizes the config option page and makes a few other changes to config docs. See a preview here: http://people.apache.org/~pwendell/config-improvements/configuration.html The biggest changes are: 1. The configs for the standalone master/workers were moved to the standalone page and out of the general config doc. 2. SPARK_LOCAL_DIRS was missing from the standalone docs. 3. Expanded discussion of injecting configs with spark-submit, including an example. 4. Config options were organized into the following categories: - Runtime Environment - Shuffle Behavior - Spark UI - Compression and Serialization - Execution Behavior - Networking - Scheduling - Security - Spark Streaming Author: Patrick Wendell <[email protected]> Closes #880 from pwendell/config-cleanup and squashes the following commits: 93f56c3 [Patrick Wendell] Feedback from Matei 6f66efc [Patrick Wendell] More feedback 16ae776 [Patrick Wendell] Adding back header section d9c264f [Patrick Wendell] Small fix e0c1728 [Patrick Wendell] Response to Matei's review 27d57db [Patrick Wendell] Reverting changes to index.html (covered in #896) e230ef9 [Patrick Wendell] Merge remote-tracking branch 'apache/master' into config-cleanup a374369 [Patrick Wendell] Line wrapping fixes fdff7fc [Patrick Wendell] Merge remote-tracking branch 'apache/master' into config-cleanup 3289ea4 [Patrick Wendell] Pulling in changes from #856 106ee31 [Patrick Wendell] Small link fix f7e79bc [Patrick Wendell] Re-organizing config options. 54b184d [Patrick Wendell] Adding standalone configs to the standalone page 592e94a [Patrick Wendell] Stash 29b5446 [Patrick Wendell] Better discussion of spark-submit in configuration docs 2d719ef [Patrick Wendell] Small fix 4af9e07 [Patrick Wendell] Adding SPARK_LOCAL_DIRS docs 204b248 [Patrick Wendell] Small fixes (cherry picked from commit 7801d44) Signed-off-by: Patrick Wendell <[email protected]>

AmplabJenkins · 2014-05-28T23:28:09Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-28T23:28:09Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15267/

This PR improves and organizes the config option page and makes a few other changes to config docs. See a preview here: http://people.apache.org/~pwendell/config-improvements/configuration.html The biggest changes are: 1. The configs for the standalone master/workers were moved to the standalone page and out of the general config doc. 2. SPARK_LOCAL_DIRS was missing from the standalone docs. 3. Expanded discussion of injecting configs with spark-submit, including an example. 4. Config options were organized into the following categories: - Runtime Environment - Shuffle Behavior - Spark UI - Compression and Serialization - Execution Behavior - Networking - Scheduling - Security - Spark Streaming Author: Patrick Wendell <[email protected]> Closes apache#880 from pwendell/config-cleanup and squashes the following commits: 93f56c3 [Patrick Wendell] Feedback from Matei 6f66efc [Patrick Wendell] More feedback 16ae776 [Patrick Wendell] Adding back header section d9c264f [Patrick Wendell] Small fix e0c1728 [Patrick Wendell] Response to Matei's review 27d57db [Patrick Wendell] Reverting changes to index.html (covered in apache#896) e230ef9 [Patrick Wendell] Merge remote-tracking branch 'apache/master' into config-cleanup a374369 [Patrick Wendell] Line wrapping fixes fdff7fc [Patrick Wendell] Merge remote-tracking branch 'apache/master' into config-cleanup 3289ea4 [Patrick Wendell] Pulling in changes from apache#856 106ee31 [Patrick Wendell] Small link fix f7e79bc [Patrick Wendell] Re-organizing config options. 54b184d [Patrick Wendell] Adding standalone configs to the standalone page 592e94a [Patrick Wendell] Stash 29b5446 [Patrick Wendell] Better discussion of spark-submit in configuration docs 2d719ef [Patrick Wendell] Small fix 4af9e07 [Patrick Wendell] Adding SPARK_LOCAL_DIRS docs 204b248 [Patrick Wendell] Small fixes

…llable information (#880)

pwendell added 7 commits May 25, 2014 21:33

Small fixes

204b248

Adding SPARK_LOCAL_DIRS docs

4af9e07

Small fix

2d719ef

Better discussion of spark-submit in configuration docs

29b5446

Stash

592e94a

Adding standalone configs to the standalone page

54b184d

Re-organizing config options.

f7e79bc

This uses the following categories: - Runtime Environment - Shuffle Behavior - Spark UI - Compression and Serialization - Execution Behavior - Networking - Scheduling - Security - Spark Streaming

pwendell changed the title ~~Organize configuration options~~ Organize configuration docs May 26, 2014

Small link fix

106ee31

Pulling in changes from apache#856

3289ea4

Merge remote-tracking branch 'apache/master' into config-cleanup

fdff7fc

Conflicts: docs/configuration.md

Line wrapping fixes

a374369

ash211 reviewed May 28, 2014
View reviewed changes

pwendell mentioned this pull request May 28, 2014

Feat kryo max buffersize #735

Closed

pwendell added 6 commits May 28, 2014 11:18

Merge remote-tracking branch 'apache/master' into config-cleanup

e230ef9

Reverting changes to index.html (covered in apache#896)

27d57db

Response to Matei's review

e0c1728

Small fix

d9c264f

Adding back header section

16ae776

More feedback

6f66efc

mateiz reviewed May 28, 2014
View reviewed changes

docs/configuration.md Outdated

Copy link

Contributor

mateiz May 28, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should say YARN

Feedback from Matei

93f56c3

asfgit closed this in 7801d44 May 28, 2014

wangyum pushed a commit that referenced this pull request May 26, 2023

[CARMEL-5867] ShowCreateTableCommand and DescribeTableCommand miss nu…

1578ee2

…llable information (#880)

Organize configuration docs #880

Organize configuration docs #880

Uh oh!

Conversation

pwendell commented May 26, 2014

Uh oh!

AmplabJenkins commented May 26, 2014

Uh oh!

AmplabJenkins commented May 26, 2014

Uh oh!

AmplabJenkins commented May 26, 2014

Uh oh!

AmplabJenkins commented May 26, 2014

Uh oh!

ash211 commented May 26, 2014

Uh oh!

pwendell commented May 26, 2014

Uh oh!

pwendell commented May 26, 2014

Uh oh!

AmplabJenkins commented May 26, 2014

Uh oh!

AmplabJenkins commented May 26, 2014

Uh oh!

AmplabJenkins commented May 26, 2014

Uh oh!

AmplabJenkins commented May 26, 2014

Uh oh!

AmplabJenkins commented May 26, 2014

Uh oh!

AmplabJenkins commented May 26, 2014

Uh oh!

pwendell commented May 26, 2014

Uh oh!

AmplabJenkins commented May 26, 2014

Uh oh!

AmplabJenkins commented May 26, 2014

Uh oh!

AmplabJenkins commented May 26, 2014

Uh oh!

AmplabJenkins commented May 26, 2014

Uh oh!

ash211 May 28, 2014

Choose a reason for hiding this comment

Uh oh!

pwendell May 28, 2014

Choose a reason for hiding this comment

Uh oh!

ash211 commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

mateiz May 28, 2014

Choose a reason for hiding this comment

Uh oh!

mateiz commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

pwendell commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!