[SPARK-12077] [SQL] change the default plan for single distinct

[davies]

Use try to match the behavior for single distinct aggregation with Spark 1.5, but that's not scalable, we should be robust by default, have a flag to address performance regression for low cardinality aggregation.

cc @yhuai @nongli

[SparkQA]

Test build #46994 has finished for PR 10075 at commit 192a04d.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

I actually prefer the old name better. Two years from now, I'm not sure how much context we will have about "1.5".

[davies]

After the discussion with @nongli @yhuai , we agreed that the only reason we have this flag here is to address the concern about performance regression for single distinct on low cardinality aggregation since 1.5, also this is the way Oracle usually did.

[rxin]

I think the big difference is that Oracle releases every x years, while we release every 3 month...

[davies]

But we are doing the same thing as maintaining compatibility and no (much) regressions, otherwise we should just remove this flag (it's not public right now).

[yhuai]

LGTM pending jenkins.

[SparkQA]

Test build #2137 has finished for PR 10075 at commit 192a04d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #47023 has finished for PR 10075 at commit 0a237b9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #2142 has finished for PR 10075 at commit 893b327.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

LGTM. Merging to master and branch 1.6.