[SPARK-12212][ML][DOC] Clarifies the difference between spark.ml, spark.mllib and mllib in the documentation.

[thunterdb]

Replaces a number of occurences of MLlib in the documentation that were meant to refer to the spark.mllib package instead. It should clarify for new users the difference between spark.mllib (the package) and MLlib (the umbrella project for ML in spark).

It also removes some files that I forgot to delete with #10207

[jkbradley]

Early comment: Maybe we should reinstate all existing .md files but replace the contents with redirection links to the relevant new parts of the guide. Websites might have links which will be broken by these changes. I guess that should include files removed in [https://github.com//pull/10207] too.

[SparkQA]

Test build #47461 has finished for PR 10234 at commit 42bf337.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * * more functionality for random forests: estimates of feature importance, as well as the predicted probability of each class (a.k.a. class conditional probabilities) for classification.\n

[jkbradley]

Related to my earlier comment, it might be good to rename the ml-intro file to ml-guide; there isn't a need to rename it.

[jkbradley]

"DataFrame" (F capitalized) (elsewhere too)

[thunterdb]

Corrected everywhere in the documentation

[jkbradley]

Done with review.
One final comment: I'd prefer adding spark.ml or spark.mllib to the end of both the title and displayTitle. I like having it in displayTitle since that provides the clearest signal about which page the user is viewing.

[thunterdb]

@jkbradley I agree with you. Note that when you use the doc, the current section appears in bold in the side menu (and the submenu gets expandend). This is why I did not include it again, but I do not have a strong opinion on this point:

[jkbradley]

OK then I'll still vote for including it : )

[srowen]

spark.mllib right?

[jkbradley]

Not this file; the "ml-" prefix ones are for spark.ml. (It's true the functionality is almost the same currently, but it's a bit different and will diverge more.)

[srowen]

I see the purpose now. It was the old MLlib text, but a lot of it still applies. The distinction is removed.

[thunterdb]

@jkbradley done with changes, let me know what you think.

[SparkQA]

Test build #47530 has finished for PR 10234 at commit 6d66645.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * * more functionality for random forests: estimates of feature importance, as well as the predicted probability of each class (a.k.a. class conditional probabilities) for classification.\n

[jkbradley]

In addition to moving ml-intro back to ml-guide, it'd be nice if the sidebar had links back to the main spark.ml and spark.mllib pages. That could be done in a separate JIRA/PR, if you prefer.

[thunterdb]

@jkbradley done:

[SparkQA]

Test build #47535 has finished for PR 10234 at commit c75a5ca.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * * more functionality for random forests: estimates of feature importance, as well as the predicted probability of each class (a.k.a. class conditional probabilities) for classification.\n

[SparkQA]

Test build #47537 has finished for PR 10234 at commit 8432ac9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * * more functionality for random forests: estimates of feature importance, as well as the predicted probability of each class (a.k.a. class conditional probabilities) for classification.\n

[jkbradley]

This doc should now be a redirect to the ml-classification-regression.html#survival-regression section.

Also, it looks like some of the math renders incorrectly, but let's fix that in a follow-up.

[jkbradley]

That's the only remaining issue I found. I checked against the Spark 1.5 doc links as well.

[jkbradley]

LGTM pending tests. Thanks! (not sure what the deal is with survival regression math; I probably am missing some library on my computer.)

[SparkQA]

Test build #47540 has finished for PR 10234 at commit 68f7e53.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * * more functionality for random forests: estimates of feature importance, as well as the predicted probability of each class (a.k.a. class conditional probabilities) for classification.\n

[jkbradley]

Merging with master and branch-1.6
Thank you!