[SPARK-7140][MLLIB] only scan the first 16 entries in Vector.hashCode

[mengxr]

The Python SerDe calls Object.hashCode, which is very expensive for Vectors. It is not necessary to scan the whole vector, especially for large ones. In this PR, we only scan the first 16 nonzeros. @srowen

[rxin]

you don't mean to have a case here do you? just a closure not partial function

[SparkQA]

Test build #30951 has finished for PR 5697 at commit 1ebad60.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch adds the following new dependencies:
  • tachyon-0.6.4.jar
  • tachyon-client-0.6.4.jar
• This patch removes the following dependencies:
  • tachyon-0.5.0.jar
  • tachyon-client-0.5.0.jar

[jkbradley]

Should SparseVector use the first 16 non-zero values instead of the first 16? For sparse data, it would not be surprising for a bunch of vectors to have all-zeros in the first 16 elements.

[mengxr]

For dense vectors, it may contains many zeros at the beginning, and then the cost is still high if we look for the first 16 nonzeros. The hashCode is not used in any MLlib code. This is just to reduce the overhead of Pyrolite serializer.

[jkbradley]

OK, sounds fine. Curious: Why do DenseVector and SparseVector need to implement the same hash code? I was thinking DenseVectors could stay as they are (the first 16 elements), while SparseVectors could look at the first 16 non-zeros to reduce collisions. But I'm not familiar with this code.

[mengxr]

This is a contract in Java (http://en.wikipedia.org/wiki/Java_hashCode()):

The general contract for overridden implementations of this method is that they behave in a way consistent with the same object's equals() method: that a given object must consistently report the same hash value (unless it is changed so that the new version is no longer considered "equal" to the old), and that two objects which equals() says are equal must report the same hash value.

So sv == dv -> sv.## == dv.##.

[SparkQA]

Test build #712 has finished for PR 5697 at commit 2abc86d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[mengxr]

Merged into master and branch-1.3.