[SPARK-13011] K-means wrapper in SparkR

[yinxusen]

https://issues.apache.org/jira/browse/SPARK-13011

[SparkQA]

Test build #50945 has finished for PR 11124 at commit ca0ea4a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[felixcheung]

shall this match the signature of stats::kmeans in R? It's pretty close already, but it might be preferable to match it
https://stat.ethz.ch/R-manual/R-devel/library/stats/html/kmeans.html

[yinxusen]

I don't think we can make the signature the same as stats:kmeans for now, since KMeans in MLlib doesn't provide the same interface with R's. E.g. KMeans in MLlib doesn't support warm-start so the centers has only to be a "numeric", and it doesn't have arguments like nstart and trace. Refer to the implementation of glm in SparkR, which also cannot match the signature with R's 100%.

[mengxr]

The current signature looks okay to me. Just need to make sure we don't shadow R's own kmeans.

[yinxusen]

I'll add test

[mengxr]

@yinxusen I made one pass. Could you update the PR and implement both summary and fitted? Thanks!

[yinxusen]

@mengxr I am on my way implementing summary. The summary of stats:kmeans contains "cluster" "centers" "totss" "withinss" "tot.withinss" "betweenss" "size" "iter" "ifault". However I think there is no need to implement all of them in this PR. I choose "cluster", "centers" and "size" in this version. What do you think?

[mengxr]

I checked R's output. It seems that summary on a k-means model doesn't provide much information:

summary(model)
             Length Class  Mode   
cluster      3      -none- numeric
centers      2      -none- numeric
totss        1      -none- numeric
withinss     2      -none- numeric
tot.withinss 1      -none- numeric
betweenss    1      -none- numeric
size         2      -none- numeric
iter         1      -none- numeric
ifault       1      -none- numeric

It seems like summary is not implemented for k-means model. So I would just implement model$centers and fitted(model, method=c("classes")). Note that we don't provide cluster (cluster assignments) and size (cluster sizes) in Scala.

[SparkQA]

Test build #51249 has finished for PR 11124 at commit 44c7595.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sun-rui]

typo: cache

[yinxusen]

@mengxr I add two summary variables cluster and size in KMeansModel to resemble counterparts in R's kmeans. IMHO even though summary(kmeans.model) looks incompleted, the kmeans.model itself contains those variables. I add these two first.

[yinxusen]

test it please

[SparkQA]

Test build #51253 has finished for PR 11124 at commit ecb2850.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

@yinxusen Could you resolve merge conflicts?

[yinxusen]

@mengxr, do it now.

On Mon, Feb 22, 2016 at 11:57 PM, Xiangrui Meng [email protected]
wrote:

@yinxusen https://github.com/yinxusen Could you resolve conflicts?

—
Reply to this email directly or view it on GitHub
#11124 (comment).

Cheers

Xusen Yin (尹绪森)
LinkedIn: https://cn.linkedin.com/in/xusenyin

[yinxusen]

test it please

[SparkQA]

Test build #51751 has finished for PR 11124 at commit 2276323.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

LGTM. Merged into master. Thanks!