[SPARK-8186][SPARK-8187][SPARK-8194][SPARK-8198][SQL] functions: date_add, date_sub, add_months, months_between, time-interval calculation #7589

adrian-wang · 2015-07-22T07:04:35Z

This subsumes #6782

adrian-wang · 2015-07-22T07:04:55Z

rxin · 2015-07-22T07:08:34Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala

i think we want to support:

timestamp + interval returns timestamp
date + interval returns date
date + int returns date ?

OK, will update today. BTW, How do we tell the difference if we are doing
string + interval/int,
should we just take strings as date?

string can get implicitly cast to date or timestamp, can't it?

Yes, but then we have to decide it is date or timestamp, and that need to be done at run time.

i think we can just put timestamp first, and as a result implicitly cast string to timestamp.

I think date_add('2015-07-22', 1) return '2015-07-23 00:00:00' is not so natural...

I just checked with hive 1.2.
hive> select date_add("2015-01-01 00:11:22", 2);
2015-01-03
hive> select date_add(cast("2015-01-01 00:11:22" as timestamp), 2);
2015-01-03

shall we use the same pattern?

date_add is used for adding days, where does hive support timestamp + interval? in + or another special operator?

hive and oracle use + sign
select birthday + interval 3 days from xxx;

OK that makes sense. Let's use date for string then.

SparkQA · 2015-07-22T07:11:26Z

Test build #38051 has finished for PR 7589 at commit c506661.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DateAdd(startDate: Expression, days: Expression)
- case class DateSub(startDate: Expression, days: Expression)

SparkQA · 2015-07-22T14:36:07Z

Test build #38078 has finished for PR 7589 at commit 1a68e03.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DateAdd(startDate: Expression, days: Expression)
- case class DateSub(startDate: Expression, days: Expression)
- case class TimeAdd(start: Expression, interval: Expression)
- case class TimeSub(start: Expression, interval: Expression)

adrian-wang · 2015-07-22T14:43:29Z

@rxin On my second thought, I think we should keep date_add and date_sub as simple as it should be. When it comes to Datetime IntervalType computation, we need another node specially for this. Here are my considerations:

hive 1.2 supports interval type, but still keep date_add and date_sub simple.
2, use another node can keep the registered function the same as most database systems.
for strings, it is hard to decide whether it is a date or timestamp, this would be troublesome. For most databases, we use date '2015-01-01' + interval XXXX to do this calculation. so the sql engine could know how we should take care of the string here.

And I have several other TODOS for the simple POC shown here, as follows:

we need to let us be aware of the precision of the interval type. For the calculation here, we would return DateType if we are add a Date and an Year-Month Interval, otherwise we should return TimestampType. For POC shown here, I just use TimestampType as the general returning type. Maybe we need two interval types here.
I need to get add support in parser. to translate the shiptime + interval XXXX into corresponding logical node.

2-1. the left operand of timestamp '2015-01-01 11:22:33' + interval XXXX should be translated into a Cast that cast string into corresponding type.

2-2. Then we translate the calculation to a TimeAdd or a TimeSub.

I think the whole stuff deserves another PR.
cc @chenghao-intel

SparkQA · 2015-07-22T18:05:06Z

Test build #1163 has finished for PR 7589 at commit 1a68e03.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DateAdd(startDate: Expression, days: Expression)
- case class DateSub(startDate: Expression, days: Expression)
- case class TimeAdd(start: Expression, interval: Expression)
- case class TimeSub(start: Expression, interval: Expression)

rxin · 2015-07-22T20:05:58Z

Let's cast string to "date". not "timestamp" then.

We should probably also add an expression for date/timestamp + interval. In that case, I don't think we want to do implicit type cast.

adrian-wang · 2015-07-23T00:47:46Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala

@rxin I did implement the expression for add date/timestamp and interval

SparkQA · 2015-07-24T17:10:00Z

Test build #38374 has finished for PR 7589 at commit 87c4b77.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DateAdd(startDate: Expression, days: Expression)
- case class DateSub(startDate: Expression, days: Expression)
- case class TimeAdd(left: Expression, right: Expression)
- case class TimeSub(left: Expression, right: Expression)
- case class AddMonths(left: Expression, right: Expression)
- case class MonthsBetween(left: Expression, right: Expression)

SparkQA · 2015-07-24T17:20:11Z

Test build #38376 has finished for PR 7589 at commit 7fcd107.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DateAdd(startDate: Expression, days: Expression)
- case class DateSub(startDate: Expression, days: Expression)
- case class TimeAdd(left: Expression, right: Expression)
- case class TimeSub(left: Expression, right: Expression)
- case class AddMonths(left: Expression, right: Expression)
- case class MonthsBetween(left: Expression, right: Expression)

SparkQA · 2015-07-24T17:49:46Z

Test build #38380 has finished for PR 7589 at commit 42df486.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DateAdd(startDate: Expression, days: Expression)
- case class DateSub(startDate: Expression, days: Expression)
- case class TimeAdd(left: Expression, right: Expression)
- case class TimeSub(left: Expression, right: Expression)
- case class AddMonths(left: Expression, right: Expression)
- case class MonthsBetween(left: Expression, right: Expression)

adrian-wang · 2015-07-24T18:37:40Z

retest this please.

SparkQA · 2015-07-24T20:17:15Z

Test build #38391 has finished for PR 7589 at commit e8a639a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DateAdd(startDate: Expression, days: Expression)
- case class DateSub(startDate: Expression, days: Expression)
- case class TimeAdd(left: Expression, right: Expression)
- case class TimeSub(left: Expression, right: Expression)
- case class AddMonths(left: Expression, right: Expression)
- case class MonthsBetween(left: Expression, right: Expression)

SparkQA · 2015-07-24T20:28:01Z

Test build #92 has finished for PR 7589 at commit e8a639a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DateAdd(startDate: Expression, days: Expression)
- case class DateSub(startDate: Expression, days: Expression)
- case class TimeAdd(left: Expression, right: Expression)
- case class TimeSub(left: Expression, right: Expression)
- case class AddMonths(left: Expression, right: Expression)
- case class MonthsBetween(left: Expression, right: Expression)

yjshen · 2015-07-25T02:03:23Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala

shall we make argument names consistent with comment?

yjshen · 2015-07-25T10:46:59Z

For months_between:

we should create a new method to get (year, month, day) from a date, in order to avoid repeated call getYearAndDayInYear in getMonth/getYear/getDayOfMonth.
when we fail to satisfy both the last day of its month or the same day, we should just use (time1 - time2) / (31.0 * MILLIS_PER_DAY * 1000L) to get month, no consideration of timezone at all.

adrian-wang · 2015-07-27T05:26:15Z

@yjshen I have checked with hive 1.2
hive> select months_between('2014-10-01 00:00:00', '2014-09-16 12:00:00');
0.5

but if we just use (time1 - time2) / (31.0 * MILLIS_PER_DAY * 1000L)
we will get 0.4677419354838....

SparkQA · 2015-07-27T07:10:41Z

Test build #38513 has finished for PR 7589 at commit 522e91a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DateAdd(startDate: Expression, days: Expression)
- case class DateSub(startDate: Expression, days: Expression)
- case class TimeAdd(left: Expression, right: Expression)
- case class TimeSub(left: Expression, right: Expression)
- case class AddMonths(left: Expression, right: Expression)
- case class MonthsBetween(left: Expression, right: Expression)

davies · 2015-07-28T19:45:24Z

sql/core/src/main/scala/org/apache/spark/sql/functions.scala

Update the doc

davies · 2015-07-28T21:41:58Z

@adrian-wang Thanks for working on this. Since we are getting close to 1.5 cut, there are several PRs on your side, do you mind to let me take over this one? (BTW, you will still take the credit.)

adrian-wang · 2015-07-29T03:08:05Z

@davies You can go ahead. I left several comments here, please take a look before you take over this one, thanks.

function date_add , date_sub

c506661

rxin reviewed Jul 22, 2015
View reviewed changes

poc of time interval calculation

1a68e03

adrian-wang reviewed Jul 23, 2015
View reviewed changes

function add_months, months_between and some fixes

87c4b77

adrian-wang changed the title ~~[SPARK-8186] [SPARK-8187] [SQL] datetime function: date_add, date_sub~~ [SPARK-8186][SPARK-8187][SPARK-8194][SPARK-8198][SQL] functions: date_add, date_sub, add_months, months_between, time-interval calculation Jul 24, 2015

adrian-wang force-pushed the udfdatecal branch from 7fcd107 to 42df486 Compare July 24, 2015 17:25

fix style

42df486

fix

e8a639a

yjshen reviewed Jul 25, 2015
View reviewed changes

fix

522e91a

davies reviewed Jul 28, 2015
View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/functions.scala

Copy link

Contributor

davies Jul 28, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update the doc

davies mentioned this pull request Jul 29, 2015

[SPARK-8186][SPARK-8187][SPARK-8194][SPARK-8198][SPARK-9133] [SPARK-9290] [SQL] functions: date_add, date_sub, add_months, months_between, time-interval calculation #7754

Closed

asfgit closed this in 1abf7dc Jul 30, 2015

[SPARK-8186][SPARK-8187][SPARK-8194][SPARK-8198][SQL] functions: date_add, date_sub, add_months, months_between, time-interval calculation #7589

[SPARK-8186][SPARK-8187][SPARK-8194][SPARK-8198][SQL] functions: date_add, date_sub, add_months, months_between, time-interval calculation #7589

Uh oh!

Conversation

adrian-wang commented Jul 22, 2015

Uh oh!

adrian-wang commented Jul 22, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 22, 2015

Uh oh!

SparkQA commented Jul 22, 2015

Uh oh!

adrian-wang commented Jul 22, 2015

Uh oh!

SparkQA commented Jul 22, 2015

Uh oh!

rxin commented Jul 22, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 24, 2015

Uh oh!

SparkQA commented Jul 24, 2015

Uh oh!

SparkQA commented Jul 24, 2015

Uh oh!

adrian-wang commented Jul 24, 2015

Uh oh!

SparkQA commented Jul 24, 2015

Uh oh!

SparkQA commented Jul 24, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yjshen commented Jul 25, 2015

Uh oh!

adrian-wang commented Jul 27, 2015

Uh oh!

SparkQA commented Jul 27, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davies commented Jul 28, 2015

Uh oh!

adrian-wang commented Jul 29, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants