[SPARK-7324][SQL] Add DataFrame.dropDuplicates

[kaka1992]

Similar to http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop_duplicates.html
def dropDuplicates(): DataFrame
def dropDuplicates(subset: Seq[String]): DataFrame

[AmplabJenkins]

Can one of the admins verify this patch?

[viirya]

Suppose that distinct is as same as dropDuplicates for removing duplicate rows? If they are the same, which implementation is better? GroupedData or Distinct node?

[kaka1992]

@viirya No, dropDuplicates is used to remove duplicate rows that are the same in some columns or in all columns (default) . The default version is as same as distinct.

[viirya]

You can also select subset of columns and then do distinct?

[kaka1992]

If you do this, you can't get all columns.

[kaka1992]

@viirya Please test this.

[kaka1992]

@rxin Please test this.

[rxin]

this is just

groupBy(subset : _*).agg(columns.map(columnFirst) : _*)

(you might need to take the head and then vararg the tail)

[rxin]

Jenkins, test this please.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[SparkQA]

Test build #31935 has started for PR 5870 at commit b6f1879.

[SparkQA]

Test build #31935 has finished for PR 5870 at commit b6f1879.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Merged build finished. Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31935/
Test PASSed.

[kaka1992]

@rxin Please retest this.

[rxin]

we should also check if subset.size == 0 or columns.size == 0, then simply return an empty data frame (there is one in SQLContext).

[adrian-wang]

I think keep the takeFirst parameter would make this better to understand.