[SPARK-17212][SQL] TypeCoercion supports widening conversion between DateType and TimestampType

[HyukjinKwon]

What changes were proposed in this pull request?

Currently, type-widening does not work between TimestampType and DateType.

This applies to SetOperation, Union, In, CaseWhen, Greatest, Leatest, CreateArray, CreateMap, Coalesce, NullIf, IfNull, Nvl and Nvl2, .

This PR adds the support for widening DateType to TimestampType for them.

For a simple example,

Before

Seq(Tuple2(new Timestamp(0), new Date(0))).toDF("a", "b").selectExpr("greatest(a, b)").show()

shows below:

cannot resolve 'greatest(`a`, `b`)' due to data type mismatch: The expressions should all have the same type, got GREATEST(timestamp, date)

or union as below:

val a = Seq(Tuple1(new Timestamp(0))).toDF()
val b = Seq(Tuple1(new Date(0))).toDF()
a.union(b).show()

shows below:

Union can only be performed on tables with the compatible column types. DateType <> TimestampType at the first column of the second table;

After

Seq(Tuple2(new Timestamp(0), new Date(0))).toDF("a", "b").selectExpr("greatest(a, b)").show()

shows below:

+----------------------------------------------------+
|greatest(CAST(a AS TIMESTAMP), CAST(b AS TIMESTAMP))|
+----------------------------------------------------+
|                                1969-12-31 16:00:...|
+----------------------------------------------------+

or union as below:

val a = Seq(Tuple1(new Timestamp(0))).toDF()
val b = Seq(Tuple1(new Date(0))).toDF()
a.union(b).show()

shows below:

+--------------------+
|                  _1|
+--------------------+
|1969-12-31 16:00:...|
|1969-12-31 00:00:...|
+--------------------+

How was this patch tested?

Unit tests in TypeCoercionSuite.

[SparkQA]

Test build #64345 has finished for PR 14786 at commit d49fbdb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Cc @cloud-fan and @petermaxlee could you please take a look for this one?

[cloud-fan]

Have you checked with other databases? It looks reasonable to do this but I'd like to know how other databases deal with this case.

[HyukjinKwon]

I have only checked Hive it seems it does not handle this case. I will check others and be back soon.

[HyukjinKwon]

MySQL and PostgreSQL support this

MySQL

• Greatest/leastest

mysql> SELECT GREATEST(CAST("1990-02-24 12:00:00" AS DATETIME), CAST("1990-02-25" AS DATE));
+-------------------------------------------------------------------------------+
| GREATEST(CAST("1990-02-24 12:00:00" AS DATETIME), CAST("1990-02-25" AS DATE)) |
+-------------------------------------------------------------------------------+
| 1990-02-25 00:00:00                                                           |
+-------------------------------------------------------------------------------+

• Union

mysql> SELECT CAST("1990-02-24 12:00:00" AS DATETIME) UNION SELECT CAST("1990-02-24" AS DATE);
+-----------------------------------------+
| CAST("1990-02-24 12:00:00" AS DATETIME) |
+-----------------------------------------+
| 1990-02-24 12:00:00                     |
| 1990-02-24 00:00:00                     |
+-----------------------------------------+

PostgreSQL

• Greatest/leatest

postgres=# SELECT GREATEST(CAST('1990-02-24 12:00:00' AS TIMESTAMP), CAST('1990-02-25' AS DATE));
      greatest
---------------------
 1990-02-25 00:00:00
(1 row)

• Union

postgres=# SELECT CAST('1990-02-24 12:00:00' AS TIMESTAMP) UNION SELECT CAST('1990-02-24' AS DATE);
      timestamp
---------------------
 1990-02-24 00:00:00
 1990-02-24 12:00:00
(2 rows)

[cloud-fan]

shall we put it in findTightestCommonTypeOfTwo?

[HyukjinKwon]

Ah, sure!

[SparkQA]

Test build #64411 has finished for PR 14786 at commit ab754fa.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

I think we only need this test

[HyukjinKwon]

Yeap, fixed.

[SparkQA]

Test build #64419 has finished for PR 14786 at commit d035eb3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to master!