SPARK-1565 (Addendum): Replace `run-example` with `spark-submit`. #704

pwendell · 2014-05-09T00:51:14Z

Gives a nicely formatted message to the user when run-example is run to
tell them to use spark-submit.

Gives a nicely formatted message to the user when `run-examle` is run to tell them to use `spark-submit`.

AmplabJenkins · 2014-05-09T00:52:57Z

Merged build triggered.

AmplabJenkins · 2014-05-09T00:53:04Z

Merged build started.

pwendell · 2014-05-09T00:55:25Z

cc/ @andrewor14

AmplabJenkins · 2014-05-09T01:31:55Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-09T01:31:55Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14832/

pwendell · 2014-05-09T02:23:28Z

bin/run-example

I thought about this some more, I think maybe we should just call spark-submit with the supplied master instead of telling the user this stuff. Or we could call spark submit and then print out the user how to run this.

Yes, I completely agree. We dont want the user to have to type out this more complicated stuff with library path and all. Just

bin/run-example org.apache.examples.spark.SparkPi <example params>

In fact, now that all the examples are inside spark.examples. package, we can try to make it even simpler. To run SparkPi, one should be able to just say

./bin/run-example SparkPi

That would very simple!

Great idea!

well but then you have streaming examples and mllib examples. Do we expect the user to type in millib.MovieLensALS then? I actually think the org.apache.examples.spark.SparkPi is more consistent with the rest (i.e. SparkSubmit). Maybe we should accept both.

Yea so I think if it starts with org.apache.spark.examples we would pass it through. If not, we'll prepend org.apache.spark.examples.

AmplabJenkins · 2014-05-09T04:12:57Z

Merged build triggered.

AmplabJenkins · 2014-05-09T04:13:05Z

Merged build started.

AmplabJenkins · 2014-05-09T04:17:57Z

Merged build triggered.

AmplabJenkins · 2014-05-09T04:18:05Z

Merged build started.

andrewor14 · 2014-05-09T04:18:26Z

README.md

nit: What does the notation [< ... >] mean? I think it's clearer if it's just [params]

AmplabJenkins · 2014-05-09T04:23:10Z

Merged build finished.

AmplabJenkins · 2014-05-09T04:23:10Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14838/

andrewor14 · 2014-05-09T04:23:55Z

README.md

pwendell · 2014-05-09T05:00:00Z

@andrewor14 great - thanks for the feedback!

AmplabJenkins · 2014-05-09T05:02:58Z

Merged build triggered.

AmplabJenkins · 2014-05-09T05:03:05Z

Merged build started.

AmplabJenkins · 2014-05-09T05:08:46Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-09T05:08:47Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14839/

andrewor14 · 2014-05-09T05:13:43Z

bin/run-example

oops sorry there's another one

andrewor14 · 2014-05-09T05:14:25Z

Pending one small change this LGTM

pwendell · 2014-05-09T05:26:02Z

Okay cool - I'll pull that change in when I merge it.

AmplabJenkins · 2014-05-09T05:38:10Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-05-09T05:38:10Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14841/

Gives a nicely formatted message to the user when `run-example` is run to tell them to use `spark-submit`. Author: Patrick Wendell <[email protected]> Closes #704 from pwendell/examples and squashes the following commits: 1996ee8 [Patrick Wendell] Feedback form Andrew 3eb7803 [Patrick Wendell] Suggestions from TD 2474668 [Patrick Wendell] SPARK-1565 (Addendum): Replace `run-example` with `spark-submit`. (cherry picked from commit 06b15ba) Signed-off-by: Patrick Wendell <[email protected]>

Looks like this change was accidentally committed here: apache@06b15ba but the change does not show up in the PR itself (apache#704). Other than not intending to go in with that PR, this also broke the test JavaApiSuite.repartition.

Looks like this change was accidentally committed here: 06b15ba but the change does not show up in the PR itself (#704). Other than not intending to go in with that PR, this also broke the test JavaAPISuite.repartition. Author: Aaron Davidson <[email protected]> Closes #716 from aarondav/shufflerand and squashes the following commits: b1cf70b [Aaron Davidson] SPARK-1770: Revert accidental(?) fix (cherry picked from commit 59577df) Signed-off-by: Aaron Davidson <[email protected]>

Looks like this change was accidentally committed here: 06b15ba but the change does not show up in the PR itself (#704). Other than not intending to go in with that PR, this also broke the test JavaAPISuite.repartition. Author: Aaron Davidson <[email protected]> Closes #716 from aarondav/shufflerand and squashes the following commits: b1cf70b [Aaron Davidson] SPARK-1770: Revert accidental(?) fix

Gives a nicely formatted message to the user when `run-example` is run to tell them to use `spark-submit`. Author: Patrick Wendell <[email protected]> Closes apache#704 from pwendell/examples and squashes the following commits: 1996ee8 [Patrick Wendell] Feedback form Andrew 3eb7803 [Patrick Wendell] Suggestions from TD 2474668 [Patrick Wendell] SPARK-1565 (Addendum): Replace `run-example` with `spark-submit`.

Looks like this change was accidentally committed here: apache@06b15ba but the change does not show up in the PR itself (apache#704). Other than not intending to go in with that PR, this also broke the test JavaAPISuite.repartition. Author: Aaron Davidson <[email protected]> Closes apache#716 from aarondav/shufflerand and squashes the following commits: b1cf70b [Aaron Davidson] SPARK-1770: Revert accidental(?) fix

- update secret and cm names to follow new convention Co-authored-by: Alexandr Navara <[email protected]>

SPARK-1565 (Addendum): Replace run-example with spark-submit.

2474668

Gives a nicely formatted message to the user when `run-examle` is run to tell them to use `spark-submit`.

pwendell reviewed May 9, 2014
View reviewed changes

Suggestions from TD

3eb7803

andrewor14 reviewed May 9, 2014
View reviewed changes

README.md Outdated

Copy link

Contributor

andrewor14 May 9, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: What does the notation [< ... >] mean? I think it's clearer if it's just [params]

andrewor14 reviewed May 9, 2014
View reviewed changes

README.md Outdated

Copy link

Contributor

andrewor14 May 9, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

N threads

Feedback form Andrew

1996ee8

andrewor14 reviewed May 9, 2014
View reviewed changes

bin/run-example

Copy link

Contributor

andrewor14 May 9, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops sorry there's another one

asfgit closed this in 06b15ba May 9, 2014

aarondav mentioned this pull request May 9, 2014

SPARK-1770: Revert accidental(?) fix #716

Closed

agirish pushed a commit to HPEEzmeral/apache-spark that referenced this pull request May 5, 2022

K8S-1488 (apache#680) (apache#704)

2af04ee

- update secret and cm names to follow new convention Co-authored-by: Alexandr Navara <[email protected]>

udaynpusa pushed a commit to mapr/spark that referenced this pull request Jan 30, 2024

K8S-1488 (apache#680) (apache#704)

fd5f975

- update secret and cm names to follow new convention Co-authored-by: Alexandr Navara <[email protected]>

mapr-devops pushed a commit to mapr/spark that referenced this pull request May 8, 2025

K8S-1488 (apache#680) (apache#704)

20ef799

- update secret and cm names to follow new convention Co-authored-by: Alexandr Navara <[email protected]>

turboFei pushed a commit to turboFei/spark that referenced this pull request Nov 6, 2025

[HADP-56879] Degrade unnecessary log from info to debug (apache#704)

25b586e

SPARK-1565 (Addendum): Replace run-example with spark-submit. #704

SPARK-1565 (Addendum): Replace run-example with spark-submit. #704

Uh oh!

Conversation

pwendell commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

pwendell commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pwendell commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented May 9, 2014

Uh oh!

pwendell commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

AmplabJenkins commented May 9, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

SPARK-1565 (Addendum): Replace `run-example` with `spark-submit`. #704

SPARK-1565 (Addendum): Replace `run-example` with `spark-submit`. #704