[SPARK-8176] [SPARK-8197] [SQL] function to_date/ trunc #6988

adrian-wang · 2015-06-24T13:28:13Z

No description provided.

SparkQA · 2015-06-24T13:35:01Z

Test build #35682 has finished for PR 6988 at commit 450159c.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ToDate(timestamp: Expression) extends Expression
- case class TRUNC(date: Expression, format: Expression)

adrian-wang · 2015-06-30T04:21:21Z

Close it because #6782 (comment) for now

adrian-wang · 2015-07-22T06:05:49Z

@chenghao-intel

chenghao-intel · 2015-07-22T06:15:36Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala

Probably only accept the DateType, TimestampType and the StringType, should be Seq(TypeCollection(DateType, TimestampType, StringType)).

TimestampType, StringType should all be convert into DateType.
Actually, I can add a rule in optimizer to delete this node after we have done implicit cast.

I mean we don't accept the IntegerType, LongType, do we? Hence we use the TypeCollection, and we'd better inherits from the ExpectInputTypes, instead of the ImplicitCastInputTypes.

int and long could not be cast into datetype.

Seems Hive only support date,string or timestamp, but not int, long..

if the cast does not work, we will get an exception, that's expected.

Yes, seems we can remove this expression in optimization.

If we removed this expression during optimizing, then we can remove the code gen stuff, too.

Could you add a comment says that this rely on implicit casting (from StringType and TimestampType)?

SparkQA · 2015-07-22T07:01:29Z

Test build #38034 has finished for PR 6988 at commit 0239f33.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ToDate(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Trunc(date: Expression, format: Expression)

SparkQA · 2015-07-22T07:18:43Z

Test build #38053 has finished for PR 6988 at commit 4054b65.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ToDate(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Trunc(date: Expression, format: Expression)

adrian-wang · 2015-07-22T07:20:49Z

retest this please.

SparkQA · 2015-07-22T07:21:33Z

Test build #52 has finished for PR 6988 at commit 4054b65.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ToDate(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Trunc(date: Expression, format: Expression)

SparkQA · 2015-07-22T07:26:23Z

Test build #38055 has finished for PR 6988 at commit 4054b65.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ToDate(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Trunc(date: Expression, format: Expression)

SparkQA · 2015-07-22T08:01:09Z

Test build #38057 has finished for PR 6988 at commit 4d64e09.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ToDate(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Trunc(date: Expression, format: Expression)

adrian-wang · 2015-07-22T10:50:59Z

retest this please.

SparkQA · 2015-07-22T10:51:30Z

Test build #58 has finished for PR 6988 at commit 4d64e09.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ToDate(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Trunc(date: Expression, format: Expression)

adrian-wang · 2015-07-22T10:52:41Z

@rxin something is wrong with Jenkins.

SparkQA · 2015-07-22T10:56:06Z

Test build #38069 has finished for PR 6988 at commit 4d64e09.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ToDate(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Trunc(date: Expression, format: Expression)

adrian-wang · 2015-07-22T17:11:28Z

retest this please.

SparkQA · 2015-07-22T18:34:21Z

Test build #38093 has finished for PR 6988 at commit 4d64e09.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ToDate(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Trunc(date: Expression, format: Expression)

SparkQA · 2015-07-22T18:48:45Z

Test build #63 has finished for PR 6988 at commit 4d64e09.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ToDate(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Trunc(date: Expression, format: Expression)

SparkQA · 2015-07-23T04:22:36Z

Test build #38165 has finished for PR 6988 at commit 0e7c33c.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.

adrian-wang · 2015-07-23T10:10:43Z

retest this please.

SparkQA · 2015-07-23T11:49:00Z

Test build #78 has finished for PR 6988 at commit d3faa53.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ToDate(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Trunc(date: Expression, format: Expression)

SparkQA · 2015-07-23T12:16:03Z

Test build #38214 has finished for PR 6988 at commit d3faa53.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

chenghao-intel · 2015-07-23T14:05:11Z

...catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala

Seems this will not trigger the codegen projections, you probably need to use the checkEvaluation instead.

SparkQA · 2015-07-27T08:58:57Z

Test build #38525 has finished for PR 6988 at commit ce15567.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ToDate(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Trunc(date: Expression, format: Expression)

davies · 2015-07-28T21:29:03Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala

you could override gen to call child.gen()

SparkQA · 2015-07-29T16:25:01Z

Test build #38847 has finished for PR 6988 at commit a476c5a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ToDate(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Trunc(date: Expression, format: Expression)

SparkQA · 2015-07-30T04:37:53Z

Test build #38963 has finished for PR 6988 at commit 980b092.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ToDate(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Trunc(date: Expression, format: Expression)

SparkQA · 2015-07-30T05:27:43Z

Test build #38987 has finished for PR 6988 at commit 310dd55.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ToDate(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Trunc(date: Expression, format: Expression)

adrian-wang · 2015-07-30T05:38:07Z

retest this please.

SparkQA · 2015-07-30T06:01:44Z

Test build #156 has finished for PR 6988 at commit 310dd55.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ToDate(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Trunc(date: Expression, format: Expression)

adrian-wang · 2015-07-30T06:31:22Z

retest this please.

SparkQA · 2015-07-30T07:26:16Z

Test build #38996 has finished for PR 6988 at commit 310dd55.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ToDate(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Trunc(date: Expression, format: Expression)

SparkQA · 2015-07-30T08:35:45Z

Test build #159 has finished for PR 6988 at commit 310dd55.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class ToDate(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
- case class Trunc(date: Expression, format: Expression)

davies · 2015-07-30T21:07:54Z

@adrian-wang Thanks for working on this, ti's pretty close, I will take over this one.

adrian-wang closed this Jun 30, 2015

adrian-wang reopened this Jul 22, 2015

adrian-wang force-pushed the udftodatetruc branch from 450159c to 0239f33 Compare July 22, 2015 05:28

adrian-wang changed the title ~~[WIP] [SPARK-8176] [SPARK-8197] [SQL] Udf to_date/ trunc~~ [SPARK-8176] [SPARK-8197] [SQL] function to_date/ trunc Jul 22, 2015

chenghao-intel reviewed Jul 22, 2015
View reviewed changes

chenghao-intel reviewed Jul 23, 2015
View reviewed changes

adrian-wang force-pushed the udftodatetruc branch from d3faa53 to ce15567 Compare July 27, 2015 07:17

adrian-wang force-pushed the udftodatetruc branch from ce15567 to d51d5f1 Compare July 28, 2015 07:24

davies reviewed Jul 28, 2015
View reviewed changes

function to_date, trunc

d44ea5f

adrian-wang force-pushed the udftodatetruc branch from d51d5f1 to a476c5a Compare July 29, 2015 13:30

address comments from davies

a476c5a

resolve rebase conflict

980b092

remove dup test in rebase

310dd55

davies mentioned this pull request Jul 30, 2015

[SPARK-8176] [SPARK-8197] [SQL] function to_date/ trunc #7805

Closed

asfgit closed this in 83670fc Jul 31, 2015

[SPARK-8176] [SPARK-8197] [SQL] function to_date/ trunc #6988

[SPARK-8176] [SPARK-8197] [SQL] function to_date/ trunc #6988

Uh oh!

Conversation

adrian-wang commented Jun 24, 2015

Uh oh!

SparkQA commented Jun 24, 2015

Uh oh!

adrian-wang commented Jun 30, 2015

Uh oh!

adrian-wang commented Jul 22, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 22, 2015

Uh oh!

SparkQA commented Jul 22, 2015

Uh oh!

adrian-wang commented Jul 22, 2015

Uh oh!

SparkQA commented Jul 22, 2015

Uh oh!

SparkQA commented Jul 22, 2015

Uh oh!

SparkQA commented Jul 22, 2015

Uh oh!

adrian-wang commented Jul 22, 2015

Uh oh!

SparkQA commented Jul 22, 2015

Uh oh!

adrian-wang commented Jul 22, 2015

Uh oh!

SparkQA commented Jul 22, 2015

Uh oh!

adrian-wang commented Jul 22, 2015

Uh oh!

SparkQA commented Jul 22, 2015

Uh oh!

SparkQA commented Jul 22, 2015

Uh oh!

SparkQA commented Jul 23, 2015

Uh oh!

adrian-wang commented Jul 23, 2015

Uh oh!

SparkQA commented Jul 23, 2015

Uh oh!

SparkQA commented Jul 23, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 27, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 29, 2015

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!

adrian-wang commented Jul 30, 2015

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!