[SPARK-1753 / 1773 / 1814] Update outdated docs for spark-submit, YARN, standalone etc. #701

andrewor14 · 2014-05-08T22:21:34Z

YARN

SparkPi was updated to not take in master as an argument; we should update the docs to reflect that.
The default YARN build guide should be in maven, not sbt.
This PR also adds a paragraph on steps to debug a YARN application.

Standalone

Emphasize spark-submit more. Right now it's one small paragraph preceding the legacy way of launching through org.apache.spark.deploy.Client.
The way we set configurations / environment variables according to the old docs is outdated. This needs to reflect changes introduced by the Spark configuration changes we made.

In general, this PR also adds a little more documentation on the new spark-shell, spark-submit, spark-defaults.conf etc here and there.

AmplabJenkins · 2014-05-08T22:22:57Z

Merged build triggered.

AmplabJenkins · 2014-05-08T22:23:03Z

Merged build started.

sryza · 2014-05-08T22:42:06Z

docs/running-on-yarn.md

If using --deploy-mode cluster, then --master should just be "yarn".

Ah, I didn't realize master=yarn-cluster also sets deployMode

AmplabJenkins · 2014-05-08T23:12:42Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-08T23:12:42Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14826/

AmplabJenkins · 2014-05-08T23:17:58Z

Merged build triggered.

AmplabJenkins · 2014-05-08T23:18:04Z

Merged build started.

AmplabJenkins · 2014-05-08T23:57:06Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-08T23:57:06Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14827/

sryza · 2014-05-09T01:07:08Z

docs/running-on-yarn.md

Nit: per-container should be hyphenated

This isn't available to all users. This only applies if you are running your own cluster and have control over nodemanager settings. I believe it also requires a nodemanager restart. on a hosted cluster you won't be able to change this so I think we should add something about that.

Good point, will do.

sryza · 2014-05-09T01:07:33Z

Added a minor nit. Otherwise, this looks good to me.

Conflicts: docs/running-on-yarn.md

AmplabJenkins · 2014-05-09T20:57:57Z

Merged build triggered.

AmplabJenkins · 2014-05-09T20:58:07Z

Merged build started.

tgravescs · 2014-05-09T21:16:27Z

thanks for the update, the changes look good to me.

I did notice one other thing in the yarn docs and that is that we don't say what parameters spark-shell takes. I know it used to list the environment variables to use to configure but now it doesn't say how to configure and I don't see any other docs that say how to do that. I actually went to look at the code to see how it is now done. So perhaps I'll file a separate jira for that unless you know of another pr perhaps handling that?

andrewor14 · 2014-05-09T21:28:27Z

AFAIK the latest spark-shell passes its arguments directly to spark-submit, so in that sense spark-shell inherits whatever parameters spark-submit takes in. scala-programming-guide discusses this very briefly. Not sure if this was what you were expecting?

Though we should probably document that spark-shell parameters are basically spark-submit parameters, and add a link between the two.

AmplabJenkins · 2014-05-09T21:38:24Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-09T21:38:24Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14855/

tgravescs · 2014-05-09T22:04:45Z

I see where the programming guide mentions a couple of options but I think it would be good to explicitly say it since people moving from 0.9 to 1.0 might not know that.

AmplabJenkins · 2014-05-10T02:07:57Z

Merged build triggered.

AmplabJenkins · 2014-05-13T00:57:58Z

Merged build triggered.

AmplabJenkins · 2014-05-13T00:58:12Z

Merged build started.

AmplabJenkins · 2014-05-13T01:22:58Z

Merged build triggered.

AmplabJenkins · 2014-05-13T01:23:19Z

Merged build started.

AmplabJenkins · 2014-05-13T01:52:57Z

Merged build triggered.

AmplabJenkins · 2014-05-13T01:58:50Z

Merged build started.

AmplabJenkins · 2014-05-13T02:28:51Z

Merged build finished.

AmplabJenkins · 2014-05-13T02:28:51Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-13T02:28:52Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14923/

AmplabJenkins · 2014-05-13T02:28:52Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14921/

…N, standalone etc. YARN - SparkPi was updated to not take in master as an argument; we should update the docs to reflect that. - The default YARN build guide should be in maven, not sbt. - This PR also adds a paragraph on steps to debug a YARN application. Standalone - Emphasize spark-submit more. Right now it's one small paragraph preceding the legacy way of launching through `org.apache.spark.deploy.Client`. - The way we set configurations / environment variables according to the old docs is outdated. This needs to reflect changes introduced by the Spark configuration changes we made. In general, this PR also adds a little more documentation on the new spark-shell, spark-submit, spark-defaults.conf etc here and there. Author: Andrew Or <[email protected]> Closes #701 from andrewor14/yarn-docs and squashes the following commits: e2c2312 [Andrew Or] Merge in changes in #752 (SPARK-1814) 25cfe7b [Andrew Or] Merge in the warning from SPARK-1753 a8c39c5 [Andrew Or] Minor changes 336bbd9 [Andrew Or] Tabs -> spaces 4d9d8f7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs 041017a [Andrew Or] Abstract Spark submit documentation to cluster-overview.html 3cc0649 [Andrew Or] Detail how to set configurations + remove legacy instructions 5b7140a [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs 85a51fc [Andrew Or] Update run-example, spark-shell, configuration etc. c10e8c7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs 381fe32 [Andrew Or] Update docs for standalone mode 757c184 [Andrew Or] Add a note about the requirements for the debugging trick f8ca990 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs 924f04c [Andrew Or] Revert addition of --deploy-mode d5fe17b [Andrew Or] Update the YARN docs (cherry picked from commit 2ffd1ea) Signed-off-by: Patrick Wendell <[email protected]>

AmplabJenkins · 2014-05-13T03:14:01Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-13T03:14:01Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14926/

…N, standalone etc. YARN - SparkPi was updated to not take in master as an argument; we should update the docs to reflect that. - The default YARN build guide should be in maven, not sbt. - This PR also adds a paragraph on steps to debug a YARN application. Standalone - Emphasize spark-submit more. Right now it's one small paragraph preceding the legacy way of launching through `org.apache.spark.deploy.Client`. - The way we set configurations / environment variables according to the old docs is outdated. This needs to reflect changes introduced by the Spark configuration changes we made. In general, this PR also adds a little more documentation on the new spark-shell, spark-submit, spark-defaults.conf etc here and there. Author: Andrew Or <[email protected]> Closes apache#701 from andrewor14/yarn-docs and squashes the following commits: e2c2312 [Andrew Or] Merge in changes in apache#752 (SPARK-1814) 25cfe7b [Andrew Or] Merge in the warning from SPARK-1753 a8c39c5 [Andrew Or] Minor changes 336bbd9 [Andrew Or] Tabs -> spaces 4d9d8f7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs 041017a [Andrew Or] Abstract Spark submit documentation to cluster-overview.html 3cc0649 [Andrew Or] Detail how to set configurations + remove legacy instructions 5b7140a [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs 85a51fc [Andrew Or] Update run-example, spark-shell, configuration etc. c10e8c7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs 381fe32 [Andrew Or] Update docs for standalone mode 757c184 [Andrew Or] Add a note about the requirements for the debugging trick f8ca990 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs 924f04c [Andrew Or] Revert addition of --deploy-mode d5fe17b [Andrew Or] Update the YARN docs

…tdout (apache#701) Co-authored-by: helenyugithub <[email protected]>

…pache#701) * minor fix ut * AL-8998 Fix resizing for ArrayBuffer in scala-2.12 (apache#691) * AL-8998 Fix resizing for ArrayBuffer in scala-2.12 * KE-42864 Fix vulnerability, upgrade tomcat-embed-core.version version to 9.0.81 (apache#699) --------- Co-authored-by: jiawei.li <[email protected]> Co-authored-by: Zhimin Wu <[email protected]>

…ache#701)

Update the YARN docs

d5fe17b

sryza reviewed May 8, 2014
View reviewed changes

Revert addition of --deploy-mode

924f04c

sryza reviewed May 9, 2014
View reviewed changes

andrewor14 added 3 commits May 9, 2014 12:03

Merge branch 'master' of github.com:apache/spark into yarn-docs

f8ca990

Conflicts: docs/running-on-yarn.md

Add a note about the requirements for the debugging trick

757c184

Update docs for standalone mode

381fe32

andrewor14 changed the title ~~[Docs] Update YARN docs~~ [Docs] Update YARN and standalone docs May 9, 2014

andrewor14 changed the title ~~[Docs] Update YARN and standalone docs~~ [SPARK-1773] Update YARN and standalone docs May 9, 2014

andrewor14 added 2 commits May 9, 2014 15:40

Merge branch 'master' of github.com:apache/spark into yarn-docs

c10e8c7

Update run-example, spark-shell, configuration etc.

85a51fc

andrewor14 changed the title ~~[SPARK-1773] Update YARN and standalone docs~~ [SPARK-1773] Update outdated docs for spark-submit, YARN, standalone etc. May 10, 2014

andrewor14 added 3 commits May 12, 2014 17:54

Abstract Spark submit documentation to cluster-overview.html

041017a

Merge branch 'master' of github.com:apache/spark into yarn-docs

4d9d8f7

Tabs -> spaces

336bbd9

andrewor14 added 2 commits May 12, 2014 18:18

Minor changes

a8c39c5

Merge in the warning from SPARK-1753

25cfe7b

andrewor14 mentioned this pull request May 13, 2014

[SPARK-1753] Warn about PySpark on YARN on Red Hat #682

Closed

andrewor14 changed the title ~~[SPARK-1773] Update outdated docs for spark-submit, YARN, standalone etc.~~ [SPARK-1753 / 1773] Update outdated docs for spark-submit, YARN, standalone etc. May 13, 2014

andrewor14 mentioned this pull request May 13, 2014

SPARK-1814: Splash page should include correct syntax for launching examples #752

Closed

Merge in changes in apache#752 (SPARK-1814)

e2c2312

andrewor14 changed the title ~~[SPARK-1753 / 1773] Update outdated docs for spark-submit, YARN, standalone etc.~~ [SPARK-1753 / 1773 / 1814] Update outdated docs for spark-submit, YARN, standalone etc. May 13, 2014

asfgit closed this in 2ffd1ea May 13, 2014

andrewor14 deleted the yarn-docs branch May 13, 2014 23:26

Alexis-D pushed a commit to Alexis-D/spark that referenced this pull request Nov 16, 2020

python runner takes an optional outputstream for redirecting stderr/s…

a995f2a

…tdout (apache#701) Co-authored-by: helenyugithub <[email protected]>

turboFei pushed a commit to turboFei/spark that referenced this pull request Nov 6, 2025

[HADP-56688] Add configuration for hive bucket definition support (ap…

3b95b51

…ache#701)

[SPARK-1753 / 1773 / 1814] Update outdated docs for spark-submit, YARN, standalone etc. #701

[SPARK-1753 / 1773 / 1814] Update outdated docs for spark-submit, YARN, standalone etc. #701

Uh oh!

Conversation

andrewor14 commented May 8, 2014

Uh oh!

AmplabJenkins commented May 8, 2014

Uh oh!

AmplabJenkins commented May 8, 2014

Uh oh!

sryza May 8, 2014

Choose a reason for hiding this comment

Uh oh!

andrewor14 May 8, 2014

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented May 8, 2014

Uh oh!

AmplabJenkins commented May 8, 2014

Uh oh!

AmplabJenkins commented May 8, 2014

Uh oh!

AmplabJenkins commented May 8, 2014

Uh oh!

AmplabJenkins commented May 8, 2014

Uh oh!

AmplabJenkins commented May 8, 2014

Uh oh!

sryza May 9, 2014

Choose a reason for hiding this comment

Uh oh!

tgravescs May 9, 2014

Choose a reason for hiding this comment

Uh oh!

andrewor14 May 9, 2014

Choose a reason for hiding this comment

Uh oh!

sryza commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

tgravescs commented May 9, 2014

Uh oh!

andrewor14 commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

tgravescs commented May 9, 2014

Uh oh!

AmplabJenkins commented May 10, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

Reviewers