SPARK-5570: No docs stating that `new SparkConf().set("spark.driver.memory", ...) will not work #4665

ilganeli · 2015-02-17T23:53:10Z

I've updated documentation to reflect true behavior of this setting in client vs. cluster mode.

…in client vs cluster mode

SparkQA · 2015-02-17T23:57:38Z

Test build #27663 has started for PR 4665 at commit db47595.

This patch merges cleanly.

SparkQA · 2015-02-18T01:10:46Z

Test build #27663 has finished for PR 4665 at commit db47595.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-18T01:10:49Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27663/
Test PASSed.

andrewor14 · 2015-02-18T18:36:49Z

Hey @ilganeli thanks for doing this. Can you also do this for the other spark.driver.* options? Like extra java opts, class paths etc.

ilganeli · 2015-02-18T19:32:12Z

Sure @andrewor14 , I presume their behavior is identical ?

andrewor14 · 2015-02-19T18:07:26Z

Yes, those won't work either if the user sets them through the conf.

andrewor14 · 2015-02-19T18:08:26Z

docs/configuration.md

By the way this explanation isn't exactly true. It only doesn't work if you set them through the conf.set(...) way. It still works if you put them in your conf/spark-defaults.conf file.

Got it - thanks for clarifying that. I'll make the relevant changes and re-submit a commit.

ilganeli · 2015-02-19T20:44:51Z

Hi @andrewor14 - Can you please review the language for the other driver options and let me know if you agree with it? Thanks!

SparkQA · 2015-02-19T20:47:36Z

Test build #27736 has started for PR 4665 at commit f149ba1.

This patch merges cleanly.

SparkQA · 2015-02-19T22:07:47Z

Test build #27736 has finished for PR 4665 at commit f149ba1.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-19T22:07:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27736/
Test FAILed.

ilganeli · 2015-02-19T22:32:50Z

What is a "MiMa test" failure??

srowen · 2015-02-19T22:42:12Z

MiMa is the check for changing public API signatures. This is a false positive since you didn't change any code, so don't worry about it, although a rebase might clear it up anyway.

Also, this may be purely a matter of taste, but I might have done <br/><em>Note:</em> to not put a blank line into the table boxes. Given the layout the extra whitespace makes it looks like there are extra rows somehow.

ilganeli · 2015-02-19T22:58:32Z

Got it - I wanted to break up the text. That's a good idea.

SparkQA · 2015-02-19T23:08:02Z

Test build #27752 has started for PR 4665 at commit 5889d43.

This patch merges cleanly.

andrewor14 · 2015-02-19T23:10:53Z

docs/configuration.md

what does (e.g. YARN deployment) mean? I would just rephrase this whole thing with:

<br/>Note: In client mode, this config must not be set through the <code>SparkConf</code> directly in your application, because the driver JVM has already started at that point. Instead, please set this through the <code>--driver-memory</code> command line option or in your default properties file.

Can you update other places with the wording suggested?

SparkQA · 2015-02-19T23:22:38Z

Test build #27754 has started for PR 4665 at commit 85f7a08.

This patch merges cleanly.

andrewor14 · 2015-02-19T23:25:01Z

docs/configuration.md

you forgot to replace this one too

actually, this one doesn't apply here. Instead, you should just replace this one with

This is used in cluster mode only.

andrewor14 · 2015-02-19T23:27:04Z

docs/configuration.md

same here, --driver-library-path

SparkQA · 2015-02-19T23:27:31Z

Test build #27755 has started for PR 4665 at commit a51700a.

This patch merges cleanly.

andrewor14 · 2015-02-19T23:28:54Z

Hi @ilganeli thanks for reiterating on my comments quickly. I think this is pretty close to being merged.

ilganeli · 2015-02-19T23:38:18Z

I'm happy to help :)

SparkQA · 2015-02-19T23:47:37Z

Test build #27756 has started for PR 4665 at commit 5d1c8dd.

This patch merges cleanly.

andrewor14 · 2015-02-19T23:50:36Z

Ok, LGTM I'm merging this into master and 1.3 thanks.

…emory", ...) will not work I've updated documentation to reflect true behavior of this setting in client vs. cluster mode. Author: Ilya Ganelin <[email protected]> Closes #4665 from ilganeli/SPARK-5570 and squashes the following commits: 5d1c8dd [Ilya Ganelin] Added example configuration code a51700a [Ilya Ganelin] Getting rid of extra spaces 85f7a08 [Ilya Ganelin] Reworded note 5889d43 [Ilya Ganelin] Formatting adjustment f149ba1 [Ilya Ganelin] Minor updates 1fec7a5 [Ilya Ganelin] Updated to add clarification for other driver properties db47595 [Ilya Ganelin] Slight formatting update c899564 [Ilya Ganelin] Merge remote-tracking branch 'upstream/master' into SPARK-5570 17b751d [Ilya Ganelin] Updated documentation for driver-memory to reflect its true behavior in client vs cluster mode (cherry picked from commit 6bddc40) Signed-off-by: Andrew Or <[email protected]>

SparkQA · 2015-02-20T00:22:46Z

Test build #27752 has finished for PR 4665 at commit 5889d43.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SparkJobInfo(namedtuple("SparkJobInfo", "jobId stageIds status")):
- class SparkStageInfo(namedtuple("SparkStageInfo",
- class StatusTracker(object):

AmplabJenkins · 2015-02-20T00:22:50Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27752/
Test PASSed.

SparkQA · 2015-02-20T00:25:06Z

Test build #27755 has finished for PR 4665 at commit a51700a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-20T00:25:10Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27755/
Test FAILed.

SparkQA · 2015-02-20T00:40:59Z

Test build #27754 has finished for PR 4665 at commit 85f7a08.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-20T00:41:03Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27754/
Test PASSed.

SparkQA · 2015-02-20T01:10:41Z

Test build #27756 has finished for PR 4665 at commit 5d1c8dd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-20T01:10:45Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27756/
Test PASSed.

Ilya Ganelin added 3 commits February 17, 2015 15:48

Updated documentation for driver-memory to reflect its true behavior …

17b751d

…in client vs cluster mode

Merge remote-tracking branch 'upstream/master' into SPARK-5570

c899564

Slight formatting update

db47595

andrewor14 reviewed Feb 19, 2015
View reviewed changes

Updated to add clarification for other driver properties

1fec7a5

Minor updates

f149ba1

Formatting adjustment

5889d43

andrewor14 reviewed Feb 19, 2015
View reviewed changes

Reworded note

85f7a08

Getting rid of extra spaces

a51700a

andrewor14 reviewed Feb 19, 2015
View reviewed changes

docs/configuration.md Outdated

Copy link

Contributor

andrewor14 Feb 19, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here, --driver-library-path

Added example configuration code

5d1c8dd

asfgit closed this in 6bddc40 Feb 19, 2015

SPARK-5570: No docs stating that `new SparkConf().set("spark.driver.memory", ...) will not work #4665

SPARK-5570: No docs stating that `new SparkConf().set("spark.driver.memory", ...) will not work #4665

Uh oh!

Conversation

ilganeli commented Feb 17, 2015

Uh oh!

SparkQA commented Feb 17, 2015

Uh oh!

SparkQA commented Feb 18, 2015

Uh oh!

AmplabJenkins commented Feb 18, 2015

Uh oh!

andrewor14 commented Feb 18, 2015

Uh oh!

ilganeli commented Feb 18, 2015

Uh oh!

andrewor14 commented Feb 19, 2015

Uh oh!

andrewor14 Feb 19, 2015

Choose a reason for hiding this comment

Uh oh!

ilganeli Feb 19, 2015

Choose a reason for hiding this comment

Uh oh!

ilganeli commented Feb 19, 2015

Uh oh!

SparkQA commented Feb 19, 2015

Uh oh!

SparkQA commented Feb 19, 2015

Uh oh!

AmplabJenkins commented Feb 19, 2015

Uh oh!

ilganeli commented Feb 19, 2015

Uh oh!

srowen commented Feb 19, 2015

Uh oh!

ilganeli commented Feb 19, 2015

Uh oh!

SparkQA commented Feb 19, 2015

Uh oh!

andrewor14 Feb 19, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Feb 19, 2015

Uh oh!

andrewor14 Feb 19, 2015

Choose a reason for hiding this comment

Uh oh!

andrewor14 Feb 19, 2015

Choose a reason for hiding this comment

Uh oh!

andrewor14 Feb 19, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Feb 19, 2015

Uh oh!

andrewor14 commented Feb 19, 2015

Uh oh!

ilganeli commented Feb 19, 2015

Uh oh!

SparkQA commented Feb 19, 2015

Uh oh!

andrewor14 commented Feb 19, 2015

Uh oh!

SparkQA commented Feb 20, 2015

Uh oh!

AmplabJenkins commented Feb 20, 2015

Uh oh!

SparkQA commented Feb 20, 2015

Uh oh!

AmplabJenkins commented Feb 20, 2015

Uh oh!

SparkQA commented Feb 20, 2015

Uh oh!

AmplabJenkins commented Feb 20, 2015

Uh oh!

SparkQA commented Feb 20, 2015

Uh oh!

AmplabJenkins commented Feb 20, 2015

Uh oh!