SPARK-4230. Doc for spark.default.parallelism is incorrect

[sryza]

No description provided.

[SparkQA]

Test build #22921 has started for PR 3107 at commit 14ca79b.

• This patch merges cleanly.

[SparkQA]

Test build #22921 has finished for PR 3107 at commit 14ca79b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22921/
Test PASSed.

[pwendell]

Should this say "number of shuffle partitions" - it's slightly weird to me to say "output" when this refers to something that is totally internal to Spark - it's output on the map side but input on he read side. In other cases I think output tends to mean things like saving as HDFS data, etc.

[sryza]

My thinking was that Spark's APIs have no mention of the concept of a "shuffle partition" (e.g. the term is referenced nowhere on https://spark.apache.org/docs/latest/programming-guide.html), but even novice Spark users are meant to understand that every transformation has input and output RDDs and that every RDD has a number of partitions.

Maybe "Default number of partitions for the RDDs produced by operations like ..."?

[pwendell]

Ah I see - what about "Default number of partitions in RDD's returned by join, reduceByKey..."

[pwendell]

Had some minor wording questions.

[SparkQA]

Test build #23129 has started for PR 3107 at commit 37a1d19.

• This patch merges cleanly.

[SparkQA]

Test build #23129 has finished for PR 3107 at commit 37a1d19.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23129/
Test FAILed.

[sryza]

Test failure looks unrelated

[pwendell]

LG - pulling it in.