SPARK-1818 Freshen Mesos documentation #756

ash211 · 2014-05-13T06:48:46Z

Place more emphasis on using precompiled binary versions of Spark and Mesos
instead of encouraging the reader to compile from source.

Place more emphasis on using precompiled binary versions of Spark and Mesos instead of encouraging the reader to compile from source.

AmplabJenkins · 2014-05-13T06:52:57Z

Merged build triggered.

AmplabJenkins · 2014-05-13T06:53:07Z

Merged build started.

pwendell · 2014-05-13T07:03:46Z

docs/running-on-mesos.md

Incidentally I think MESOS_VERSION needs to be updated to 0.18 - mind doing that?

Updated to 0.18.1 per the pom.xml

AmplabJenkins · 2014-05-13T07:12:57Z

Merged build triggered.

AmplabJenkins · 2014-05-13T07:13:07Z

Merged build started.

AmplabJenkins · 2014-05-13T07:32:12Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-13T07:32:13Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14936/

pwendell · 2014-05-13T07:40:29Z

docs/running-on-mesos.md

Period at the end of this sentence?

pwendell · 2014-05-13T07:44:26Z

Made a few surface level comments - this is a great and much-needed update. Thanks!

AmplabJenkins · 2014-05-13T07:52:35Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-13T07:52:35Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14937/

At least that's how I read SparkILoop#createSparkContext()

ash211 · 2014-05-13T08:17:15Z

Made changes to cover Patrick's suggestions.

I had some trouble getting LZO libraries accessible from the Mesos executor and never actually got that working. Do you know if there's a way to add additional .jars to the Mesos executors, or do all .jars need to be bundled in the distribution tarball downloaded from e.g. HDFS?

If so I'd like to add that to the documentation too.

AmplabJenkins · 2014-05-13T08:17:57Z

Merged build triggered.

AmplabJenkins · 2014-05-13T08:18:02Z

Merged build started.

ash211 · 2014-05-13T08:44:34Z

cc: @ceteri -- we're updating the Spark documentation for running on Mesos. Given your expertise with this setup (I believe you did the video and wrote the docs below), would you mind taking a look at this documentation refresh?

This will likely end up being the documentation for the next Spark release and I'd like to get it in a great state for the big 1.0

https://www.youtube.com/watch?v=KVWMhIeKM_A
http://mesosphere.io/learn/run-spark-on-mesos

Thanks!
Andrew

AmplabJenkins · 2014-05-13T08:57:04Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-13T08:57:04Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14938/

maasg · 2014-05-13T11:44:39Z

docs/running-on-mesos.md

I miss a chapter on "How Spark works on Mesos" that helps understanding the need of the assembly in a reachable location by the slaves and how the spark driver program becomes the 'scheduler' in Mesos terms and the tasks get delivered to the executors.

maasg · 2014-05-13T11:52:35Z

Great work.
I'd love to see some more background of the dynamics of Spark running on Mesos. It has been a tough learning experience to get our Spark + Spark Streaming running on a Mesos cluster, and I think the docs can make the life easier for 'future generations' :-)

ash211 · 2014-05-14T05:49:22Z

Hi @maasg thanks for the review!

I added a section on troubleshooting and debugging, and also a quick overview of how Spark and Mesos interaction in a How it works section.

Do those start to cover what you were looking for as an addition?

AmplabJenkins · 2014-05-14T05:52:57Z

Merged build triggered.

AmplabJenkins · 2014-05-14T05:53:05Z

Merged build started.

AmplabJenkins · 2014-05-14T05:54:23Z

Merged build finished.

AmplabJenkins · 2014-05-14T05:54:23Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14961/

pwendell · 2014-05-14T16:36:44Z

docs/running-on-mesos.md

I think this needs to be highlight bash. At least on my version of jekyll I get an error otherwise when compiling docs.

pwendell · 2014-05-14T16:45:20Z

I'm going to pull this in with the required build fix so it gets into a release candidate. This is a major improvement to the mesos docs. Please feel free to submit follow on work in a new PR.

Place more emphasis on using precompiled binary versions of Spark and Mesos instead of encouraging the reader to compile from source. Author: Andrew Ash <[email protected]> Closes #756 from ash211/spark-1818 and squashes the following commits: 7ef3b33 [Andrew Ash] Brief explanation of the interactions between Spark and Mesos e7dea8e [Andrew Ash] Add troubleshooting and debugging section 956362d [Andrew Ash] Don't need to pass spark.executor.uri into the spark shell de3353b [Andrew Ash] Wrap to 100char 7ebf6ef [Andrew Ash] Polish on the section on Mesos Master URLs 3dcc2c1 [Andrew Ash] Use --tgz parameter of make-distribution 41b68ed [Andrew Ash] Period at end of sentence; formatting on :5050 8bf2c53 [Andrew Ash] Update site.MESOS_VERSIOn to match /pom.xml 74f2040 [Andrew Ash] SPARK-1818 Freshen Mesos documentation (cherry picked from commit d1d41cc) Signed-off-by: Patrick Wendell <[email protected]>

ash211 · 2014-05-14T16:53:13Z

Thanks Patrick!
On May 14, 2014 9:45 AM, "Patrick Wendell" [email protected] wrote:

I'm going to pull this in with the required build fix so it gets into a
release candidate. This is a major improvement to the mesos docs. Please
feel free to submit follow on work in a new PR.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/756#issuecomment-43105912
.

maasg · 2014-05-15T15:08:17Z

Thanks for the updates! +1 (after the facts :) )

mateiz · 2014-05-16T22:14:58Z

docs/running-on-mesos.md

@ash211 @pwendell, I missed this when the PR was being reviewed, but we need to list the source instructions first. We are first and foremost an Apache project, and need to tell people how to use Apache software to get their work done. Third parties can go ahead and publish binary distributions if they want, but we can't be posting that as the preferred way to install another Apache project. We do the same thing for Hadoop distros -- we say how to build against an Apache release of Hadoop, and then we have a page on building against third-party distros.

Also I would call these "third-party packages" instead of "prebuilt packages". Again, since this is going on an Apache web page, we can't be misrepresenting the Mesos project. Bits that are not built by Apache are unfortunately third-party.

Got it. I'm starting a cleanup PR that should be in within an hour.

On Fri, May 16, 2014 at 3:17 PM, Matei Zaharia [email protected]:

In docs/running-on-mesos.md:

+# Installing Mesos
+
+Spark {{site.SPARK_VERSION}} is designed for use with Mesos {{site.MESOS_VERSION}} and does not
+require any special patches of Mesos.
+
+If you already have a Mesos cluster running, you can skip this Mesos installation step.
+
+Otherwise, installing Mesos for Spark is no different than installing Mesos for use by other
+frameworks. You can install Mesos using either prebuilt packages or by compiling from source.
+
+## Prebuilt packages
+
+The Apache Mesos project only publishes source package releases, no binary releases. But other
+third party projects publish binary releases that may be helpful in setting Mesos up.
+
+One of those is Mesosphere. To install Mesos using the binary releases provided by Mesosphere:

Also I would call these "third-party packages" instead of "prebuilt
packages". Again, since this is going on an Apache web page, we can't be
misrepresenting the Mesos project. Bits that are not built by Apache are
unfortunately third-party.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/756/files#r12761962
.

#805

On Fri, May 16, 2014 at 3:41 PM, Andrew Ash [email protected] wrote:

Got it. I'm starting a cleanup PR that should be in within an hour.

On Fri, May 16, 2014 at 3:17 PM, Matei Zaharia [email protected]:

In docs/running-on-mesos.md:

+# Installing Mesos
+
+Spark {{site.SPARK_VERSION}} is designed for use with Mesos {{site.MESOS_VERSION}} and does not
+require any special patches of Mesos.
+
+If you already have a Mesos cluster running, you can skip this Mesos installation step.
+
+Otherwise, installing Mesos for Spark is no different than installing Mesos for use by other
+frameworks. You can install Mesos using either prebuilt packages or by compiling from source.
+
+## Prebuilt packages
+
+The Apache Mesos project only publishes source package releases, no binary releases. But other
+third party projects publish binary releases that may be helpful in setting Mesos up.
+
+One of those is Mesosphere. To install Mesos using the binary releases provided by Mesosphere:

Also I would call these "third-party packages" instead of "prebuilt
packages". Again, since this is going on an Apache web page, we can't be
misrepresenting the Mesos project. Bits that are not built by Apache are
unfortunately third-party.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/756/files#r12761962
.

Place more emphasis on using precompiled binary versions of Spark and Mesos instead of encouraging the reader to compile from source. Author: Andrew Ash <[email protected]> Closes apache#756 from ash211/spark-1818 and squashes the following commits: 7ef3b33 [Andrew Ash] Brief explanation of the interactions between Spark and Mesos e7dea8e [Andrew Ash] Add troubleshooting and debugging section 956362d [Andrew Ash] Don't need to pass spark.executor.uri into the spark shell de3353b [Andrew Ash] Wrap to 100char 7ebf6ef [Andrew Ash] Polish on the section on Mesos Master URLs 3dcc2c1 [Andrew Ash] Use --tgz parameter of make-distribution 41b68ed [Andrew Ash] Period at end of sentence; formatting on :5050 8bf2c53 [Andrew Ash] Update site.MESOS_VERSIOn to match /pom.xml 74f2040 [Andrew Ash] SPARK-1818 Freshen Mesos documentation

SPARK-1818 Freshen Mesos documentation

74f2040

Place more emphasis on using precompiled binary versions of Spark and Mesos instead of encouraging the reader to compile from source.

pwendell reviewed May 13, 2014
View reviewed changes

Update site.MESOS_VERSIOn to match /pom.xml

8bf2c53

pwendell reviewed May 13, 2014
View reviewed changes

docs/running-on-mesos.md Outdated

Copy link

Contributor

pwendell May 13, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Period at the end of this sentence?

ash211 added 2 commits May 13, 2014 00:47

Period at end of sentence; formatting on :5050

41b68ed

Use --tgz parameter of make-distribution

3dcc2c1

ash211 added 3 commits May 13, 2014 01:06

Polish on the section on Mesos Master URLs

7ebf6ef

Wrap to 100char

de3353b

Don't need to pass spark.executor.uri into the spark shell

956362d

At least that's how I read SparkILoop#createSparkContext()

maasg reviewed May 13, 2014
View reviewed changes

Add troubleshooting and debugging section

e7dea8e

Brief explanation of the interactions between Spark and Mesos

7ef3b33

pwendell reviewed May 14, 2014
View reviewed changes

asfgit closed this in d1d41cc May 14, 2014

mateiz reviewed May 16, 2014
View reviewed changes

SPARK-1818 Freshen Mesos documentation #756

SPARK-1818 Freshen Mesos documentation #756

Uh oh!

Conversation

ash211 commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pwendell commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

ash211 commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

ash211 commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

AmplabJenkins commented May 13, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maasg commented May 13, 2014

Uh oh!

ash211 commented May 14, 2014

Uh oh!

AmplabJenkins commented May 14, 2014

Uh oh!

AmplabJenkins commented May 14, 2014

Uh oh!

AmplabJenkins commented May 14, 2014

Uh oh!

AmplabJenkins commented May 14, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pwendell commented May 14, 2014

Uh oh!

ash211 commented May 14, 2014

Uh oh!

maasg commented May 15, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants