[SPARK-16372][MLlib] Retag RDD to tallSkinnyQR of RowMatrix

[yinxusen]

What changes were proposed in this pull request?

The following Java code because of type erasing:

JavaRDD<Vector> rows = jsc.parallelize(...);
RowMatrix mat = new RowMatrix(rows.rdd());
QRDecomposition<RowMatrix, Matrix> result = mat.tallSkinnyQR(true);

We should use retag to restore the type to prevent the following exception:

java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to [Lorg.apache.spark.mllib.linalg.Vector;

How was this patch tested?

Java unit test

[SparkQA]

Test build #61737 has finished for PR 14051 at commit 82b4edd.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #61739 has finished for PR 14051 at commit 0acd1e0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Where does the exception actually occur? I guess I'm surprised if this is the only place this is needed.

[yinxusen]

Since it's a known Java type erased issue (https://issues.apache.org/jira/browse/SPARK-2737), I am not sure wether to fix it or not. If leaving it as is, then Java users should aware of it and retag the JavaRDD themselves. Otherwise we fix its constructors with either retaging the rows or adding a new JavaRDD constructor. However this may not be a single sample.

[srowen]

I agree with fixing it, just wonder exactly where the exception arises (not the nature of the problem; I get that) to verify this is the right place to retag. It seemed a little surprising but I assume you're right.

[yinxusen]

From my log, I can see that it arises at the glom() function. Just like the collect(), they have a similar operation (iter: Iterator[T]) => iter.toArray. So I think maybe here is the best place to call retag.

[yinxusen]

I also tried other interfaces of RowMatrix, all work good:

JavaRDD<Vector> rows = jsc.parallelize(Arrays.asList(v1, v2, v3), 1);
Matrix dm = Matrices.dense(3, 2, new double[] {1.0, 3.0, 5.0, 2.0, 4.0, 6.0});
RowMatrix mat = new RowMatrix(rows.rdd());

mat.computeGramianMatrix();
mat.columnSimilarities();
mat.columnSimilarities(0.5);
mat.computeColumnSummaryStatistics();
mat.computeCovariance();
mat.computePrincipalComponents(1);
mat.computeSVD(1, false, 1e-9);
mat.toBreeze();
mat.rows();
mat.numCols();
mat.numRows();
mat.multiply(dm);

[srowen]

Merged to master/2.0/1.6. I think it's a reasonably important bug fix.

[zsxwing]

This one broke branch 1.6. I just reverted it. Please resubmit a backport for branch 1.6.

[srowen]

@zsxwing crumbs, thanks for that. It looks reasonably sure it's related, though, I still can't quite figure out how it would cause this failure:

[error] /home/jenkins/workspace/spark-branch-1.6-compile-maven-scala-2.11/mllib/src/test/java/org/apache/spark/mllib/linalg/distributed/JavaRowMatrixSuite.java:24: error: cannot find symbol
[error] import org.apache.spark.SharedSparkSession;
[error]                        ^
[error]   symbol:   class SharedSparkSession
[error]   location: package org.apache.spark
[error] /home/jenkins/workspace/spark-branch-1.6-compile-maven-scala-2.11/mllib/src/test/java/org/apache/spark/mllib/linalg/distributed/JavaRowMatrixSuite.java:31: error: cannot find symbol
[error] public class JavaRowMatrixSuite extends SharedSparkSession {
[error]                                         ^
[error]   symbol: class SharedSparkSession
[error] /home/jenkins/workspace/spark-branch-1.6-compile-maven-scala-2.11/mllib/src/test/java/org/apache/spark/mllib/linalg/distributed/JavaRowMatrixSuite.java:39: error: cannot find symbol
[error]     JavaRDD<Vector> rows = jsc.parallelize(Arrays.asList(v1, v2, v3), 1);
[error]                            ^

Well, maybe safest to just leave this out of 1.6 in any event