SPARK-1492. Update Spark YARN docs to use spark-submit #601
Conversation
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
docs/cluster-overview.md
Outdated
There was a problem hiding this comment.
Maybe mention that spark-submit has a "--jars" option too?
There was a problem hiding this comment.
yeah I like that idea. Something like When using ./bin/spark-submit, the application jar along with those included via the --jars flag will automatically be transferred to the cluster
|
Hey @sryza thanks a bunch for this. Looking good. I built it locally and read through the doc. I noticed a few other issues with the doc that you can choose to address or not depending on if you have time. In general, I think the doc makes it seem like you need to build Spark yourself to submit a YARN job, but actually, most users should not have to do this. |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14604/ |
docs/running-on-yarn.md
Outdated
There was a problem hiding this comment.
jw - is it normal to do hdfs:/some/path and not hdfs://some/path? I think they are technically both valid URL's.
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
Thanks Sandy, I've merged this. |
Author: Sandy Ryza <[email protected]> Closes #601 from sryza/sandy-spark-1492 and squashes the following commits: 5df1634 [Sandy Ryza] Address additional comments from Patrick. be46d1f [Sandy Ryza] Address feedback from Marcelo and Patrick 867a3ea [Sandy Ryza] SPARK-1492. Update Spark YARN docs to use spark-submit (cherry picked from commit 2b961d8) Signed-off-by: Patrick Wendell <[email protected]>
Author: Sandy Ryza <[email protected]> Closes apache#601 from sryza/sandy-spark-1492 and squashes the following commits: 5df1634 [Sandy Ryza] Address additional comments from Patrick. be46d1f [Sandy Ryza] Address feedback from Marcelo and Patrick 867a3ea [Sandy Ryza] SPARK-1492. Update Spark YARN docs to use spark-submit
) Move cloud-provider-openstack jobs from vexxhost to citynetwork. lb job is still on vexxhost, since octavia is not available on citynetwork at this moment. Close-Bug: theopenlab/openlab#321
…ache#601) Co-authored-by: sibing.zhang <[email protected]>
…ache#601) Co-authored-by: sibing.zhang <[email protected]>
…ache#601) Co-authored-by: sibing.zhang <[email protected]>
4 participants
No description provided.