[SPARK-7242][SQL][MLLIB] Frequent items for DataFrames

[brkyvz]

Finding frequent items with possibly false positives, using the algorithm described in http://www.cs.umd.edu/~samir/498/karp.pdf.
public API under:

df.stat.freqItems(cols: Array[String], support: Double = 0.001): DataFrame

The output is a local DataFrame having the input column names with -freqItems appended to it. This is a single pass algorithm that may return false positives, but no false negatives.

cc @mengxr @rxin

Let's get the implementations in, I can add python API in a follow up PR.

[rxin]

does this work in java?

[brkyvz]

I think it looks like df.stat$.MODULE$.freqItems(). I don't know how we can otherwise make it df.stat.freqItems in scala.

[rxin]

take a look at how we implemented na.

[brkyvz]

aha! I like it

[rxin]

let's put this in execution.stat?

It's annoying to add a top level package because we have rules to specifically exclude existing packages.

[rxin]

I'm going to let @mengxr to comment on the actual algorithm implementation.

[mengxr]

If multiple columns are provided, shall we search the combination of them instead of each individually? For example, if I call

freqItems(Array("gender", "title"), 0.01)

I'm expecting the frequent combinations instead of each of them. The current implementation is more flexible because users can create a struct from multiple columns, and this allows to find frequent items on multiple columns in parallel. But I'm a little worried about what users expect when they call freqItems(Array("gender", "title")) @rxin

[rxin]

don't forget to add java.util.List ones

[rxin]

also make sure you add a test to the JavaDataFrameSuite

[SparkQA]

Test build #31386 has finished for PR 5799 at commit 8279d4d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[rxin]

I think it's better to just have freqItems to on a per column basis, and then I can add a struct expression to data frame so users can easily create composite columns to run freqItems on.

[SparkQA]

Test build #31392 has finished for PR 5799 at commit 482e741.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch adds the following new dependencies:
  • jaxb-api-2.2.7.jar
  • jaxb-core-2.2.7.jar
  • jaxb-impl-2.2.7.jar
  • pmml-agent-1.1.15.jar
  • pmml-model-1.1.15.jar
  • pmml-schema-1.1.15.jar
• This patch removes the following dependencies:
  • activation-1.1.jar
  • jaxb-api-2.2.2.jar
  • jaxb-impl-2.2.3-1.jar

[SparkQA]

Test build #31404 has finished for PR 5799 at commit 3a5c177.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch adds the following new dependencies:
  • jaxb-api-2.2.7.jar
  • jaxb-core-2.2.7.jar
  • jaxb-impl-2.2.7.jar
  • pmml-agent-1.1.15.jar
  • pmml-model-1.1.15.jar
  • pmml-schema-1.1.15.jar
• This patch removes the following dependencies:
  • activation-1.1.jar
  • jaxb-api-2.2.2.jar
  • jaxb-impl-2.2.3-1.jar

[SparkQA]

Test build #31423 has finished for PR 5799 at commit 0915e23.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #31426 has finished for PR 5799 at commit 39b1bba.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[mengxr]

space before {

[mengxr]

LGTM except minor inline comments.

[mengxr]

organize imports

[rxin]

I think we can just use Seq here, since Python has helper functions that can convert List into Seq.

[SparkQA]

Test build #31453 has finished for PR 5799 at commit a6ec82c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[brkyvz]

@rxin, can you merge this in please. I can follow up with the comments on the PR where I'll add Python support. This is blocking df.stat.cov. Thanks!

[rxin]

Alright - merging in master.