[SPARK-2665] [SQL] Add EqualNS & Unit Tests

[chenghao-intel]

Hive Supports the operator "<=>", which returns same result with EQUAL(=) operator for non-null operands, but returns TRUE if both are NULL, FALSE if one of the them is NULL.

[rxin]

This is really nice - passes a lot more tests. I guess we will eventually run into the problem that the unit tests without parallelization will take too long...

[rxin]

Do you mind looking into how we can parallelize the hive compatibility tests?

[SparkQA]

QA tests have started for PR 1570. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17113/consoleFull

[chenghao-intel]

Yes, I will think about how to parallelize those tests.

[marmbrus]

There is already support for running the hive tests in parallel.

Though there might be a bug in it?

[rxin]

Does our CI script use it?

[marmbrus]

Can we spell this out? (i.e. EqualsNullSafe) Since there is no conflict with Scala we can probably leave out the "To".

[chenghao-intel]

Got it. will rename it.

[marmbrus]

No, but that would be awesome.

[rxin]

Yup!

[SparkQA]

QA tests have started for PR 1570. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17116/consoleFull

[adrian-wang]

I think you forgot to white-list the test:)

[chenghao-intel]

Yes, thank you very much. Added :)

[SparkQA]

QA tests have started for PR 1570. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17117/consoleFull

[SparkQA]

QA results for PR 1570:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
case class EqualNSTo(left: Expression, right: Expression) extends BinaryComparison {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17113/consoleFull

[SparkQA]

QA results for PR 1570:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
case class EqualNullSafe(left: Expression, right: Expression) extends BinaryComparison {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17116/consoleFull

[SparkQA]

QA results for PR 1570:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
case class EqualNullSafe(left: Expression, right: Expression) extends BinaryComparison {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17117/consoleFull

[marmbrus]

orc_predicate_pushdown looks like a poorly written test to me. Both of the results returned satisfy the predicate. Really you should never have a limit clause without a ordering.

I'd move this tests from the whitelist to the blacklist (with a note about why) and delete the test files.

[chenghao-intel]

Thank you @marmbrus , it's done.

[SparkQA]

QA tests have started for PR 1570. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17149/consoleFull

[SparkQA]

QA results for PR 1570:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
case class EqualNullSafe(left: Expression, right: Expression) extends BinaryComparison {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17149/consoleFull

[marmbrus]

Thanks! I've merge this into master