[SPARK-1969][MLlib] Online summarizer APIs for mean, variance, min, and max

[dbtsai]

It basically moved the private ColumnStatisticsAggregator class from RowMatrix to public available DeveloperApi with documentation and unitests.

Changes:

1. Moved the private implementation from org.apache.spark.mllib.linalg.ColumnStatisticsAggregator to org.apache.spark.mllib.stat.MultivariateOnlineSummarizer
2. When creating OnlineSummarizer object, the number of columns is not needed in the constructor. It's determined when users add the first sample.
3. Added the APIs documentation for MultivariateOnlineSummarizer.
4. Added the unittests for MultivariateOnlineSummarizer.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[mateiz]

MultivariateStatisticalSummary is a public API -- we can't rename it arbitrarily. Why does it need to be renamed?

[dbtsai]

Since the "Statistical" in MultivariateStatisticalSummary is already in the package name as "stat", I think it worths to have a concise name. Also, most people spell the abbreviation of statistics as "stats", so I changed it from "stat" to "stats".

Since it's already a public API, I've no problem to change it back.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15407/

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15405/

[dbtsai]

Don't know why jenkins is not happy with removing "private class ColumnStatisticsAggregator(private val n: Int)". After all, it's a private class.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15408/

[mengxr]

Maybe this is a MIMA problem. Found this (from @pwendell):

https://groups.google.com/forum/#!topic/migration-manager-user/5aQ0xxsL2lU

[dbtsai]

@mengxr Get you. It's false-positive error. Do you have any comment or feedback moving it out as public api? I'm building a feature scaling api in MlUtils which depends on this. Thanks.

[mengxr]

@dbtsai The current workaround is excluding it in project/MimaExcludes.scala. Please check the examples there. At least, we need to make Jenkins happy.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[dbtsai]

k... better to have Mima exclude the private class automatically, or we can have annotation for the private class.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15492/

[AmplabJenkins]

Build triggered.

[AmplabJenkins]

Build started.

[AmplabJenkins]

Build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15504/

[mengxr]

@dbtsai About the package name, stat is the standard acronym for statistics instead of stats. Checkout the urls returned by Google:

https://www.google.com/#q=statistics+department

[mengxr]

merge or aggregate may be better than overloading add here.

[mengxr]

streaming has special meaning in spark. Change it to online?

[SparkQA]

QA tests have started for PR 955. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16558/consoleFull

[SparkQA]

QA tests have started for PR 955. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16560/consoleFull

[SparkQA]

QA tests have started for PR 955. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16561/consoleFull

[SparkQA]

QA results for PR 955:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class MultivariateOnlineSummarizer extends MultivariateStatisticalSummary with Serializable {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16558/consoleFull

[SparkQA]

QA results for PR 955:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class MultivariateOnlineSummarizer extends MultivariateStatisticalSummary with Serializable {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16560/consoleFull

[SparkQA]

QA results for PR 955:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class MultivariateOnlineSummarizer extends MultivariateStatisticalSummary with Serializable {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16561/consoleFull

[SparkQA]

QA tests have started for PR 955. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16576/consoleFull

[SparkQA]

QA results for PR 955:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class MultivariateOnlineSummarizer extends MultivariateStatisticalSummary with Serializable {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16576/consoleFull

[mengxr]

Merged. Thanks!