[SPARK-12933][SQL] Initial implementation of Count-Min sketch

[liancheng]

This PR adds an initial implementation of count min sketch, contained in a new module spark-sketch under common/sketch. The implementation is based on the CountMinSketch class in stream-lib.

As required by the design doc, spark-sketch should have no external dependency.
Two classes, Murmur3_x86_32 and Platform are copied to spark-sketch from spark-unsafe for hashing facilities. They'll also be used in the upcoming bloom filter implementation.

The following features will be added in future follow-up PRs:

• Serialization support
• DataFrame API integration

[rxin]

we should add some comment acknowledging stream-lib

[rxin]

do we also need to update the test runner to add this module?

cc @JoshRosen

[liancheng]

ClassTag is used here for creating arrays. I found using Seq can slow down test execution quite a bit.

[liancheng]

@rxin Already added sketch module to sparktestsupport per Josh's suggestion.

[SparkQA]

Test build #49811 has finished for PR 10851 at commit bb7435f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

kind of black magic...

[rxin]

Somehow there is no timing information for the test cases in this new module. Can you take a look at that? You might need to change the sbt build file.

[JoshRosen]

Oh, I forgot: you also need to update project/SparkBuild.scala to add the new submodule.

[SparkQA]

Test build #49820 has finished for PR 10851 at commit 486414d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #49824 has finished for PR 10851 at commit 92d3a52.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

an interface?

[rxin]

I'd just start with "A Count-Min sketch is a probabilistic data structure ..."

i.e. your second paragraph.

And then explain the type of data types supported.

[rxin]

declare that this could throw some exception?

[SparkQA]

Test build #49826 has finished for PR 10851 at commit 2bf907a.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #49827 has finished for PR 10851 at commit 7ea22a9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #49848 has finished for PR 10851 at commit a6e7479.

• This patch fails Java style tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #49882 has finished for PR 10851 at commit e06ff13.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[liancheng]

Had to break it down since the original pattern match somehow introduced an implicit tuple containing more than 22 fields after adding the spark-sketch module.

[SparkQA]

Test build #49923 has finished for PR 10851 at commit 4adb57a.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #49925 has finished for PR 10851 at commit 57a31e6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

we should add some comment here explaining this is just a duplicate and is put here to minimize dependency.

[liancheng]

Added comments for the duplicated Platform class and Murmur3_x86_32 class.

[rxin]

@liancheng can you make sure the generated javadocs look ok?

[liancheng]

I've checked the Javadoc, it looks good.

[rxin]

I looked at this quickly (i.e. didn't do a detail review), but changes lgtm.

[SparkQA]

Test build #2445 has started for PR 10851 at commit 1608ec9.

[SparkQA]

Test build #49929 has finished for PR 10851 at commit 65853ad.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Going to merge this. Thanks.

Would be great if @cloud-fan can take another look at the implementation.

[cloud-fan]

the CMSMergeException is a protected static class, can user catch this exception?

[liancheng]

Thanks, actually I've fixed this issue in # 10893.

[liancheng]

@rxin The constructor visibility issue would be fixed in #10893.