Proposal: clarify Scala programming guide on caching ...

[esjewett]

... with regards to saved map output. Wording taken partially from Matei Zaharia's email to the Spark user list. http://apache-spark-user-list.1001560.n3.nabble.com/performance-improvement-on-second-operation-without-caching-td5227.html

[AmplabJenkins]

Can one of the admins verify this patch?

[pwendell]

Not your change, but I think this should say "will be persisted in memory or on disk on the nodes"

[pwendell]

Oh sorry - nevermind, this is explained below and this case only refers to calling persist() without arguments.

[esjewett]

Just putting it out there: I'm not attached to any of this wording, so change away, or don't accept it. No problem either way. I just thought my question on the user list as to whether the programming guide could be updated was better stated as a pull request ;-)

[pwendell]

It's a great idea to have this here. This is a totally non-obvious fact and I think many users would like to know this.

My only thought is, would you mind moving this to the end of the "RDD Persistence" section. Also, at this point in the guide I don't think the concept of stages or jobs has been introduced. So it might be good to have something like:

Spark sometimes automatically persists intermediate state from RDD operations, even without users calling
persist() or cache(). In particular, if a shuffle happens when computing an RDD, Spark will keep the outputs
from the map side of the shuffle on disk to avoid re-computing the entire dependency graph if an RDD
 is re-used. We still recommend users call persist() if they plan to re-use an RDD iteratively.

[esjewett]

@pwendell I like your wording. Switched to use it, and moved it to the end of the "RDD Persistence" section as requested. I also updated the "RDD Operations" section with a small change so as not to imply that RDDs that aren't persist()ed will always be reprocessed.

[pwendell]

Okay I can merge this, thanks!