[SPARK-24673][SQL] scala sql function from_utc_timestamp second argument could be Column instead of String #21693
Conversation
|
See https://spark-prs.appspot.com/ ; you'll want to fix up the title of this PR. |
tmnd1991
changed the title
| * zone, and renders that time as a timestamp in UTC. For example, 'GMT+1' would yield | ||
| * '2017-07-14 01:40:00.0'. | ||
| * @group datetime_funcs | ||
| * @since 1.5.0 |
| * that time as a timestamp in the given time zone. For example, 'GMT+1' would yield | ||
| * '2017-07-14 03:40:00.0'. | ||
| * @group datetime_funcs | ||
| * @since 1.5.0 |
|
cc: @ueshin @gatorsmile |
srowen
left a comment
srowen
left a comment
There was a problem hiding this comment.
The overload seems reasonable to me.
|
Test build #4204 has finished for PR 21693 at commit
|
| * @group datetime_funcs | ||
| * @since 2.4.0 | ||
| */ | ||
| def from_utc_timestamp(ts: Column, tz: Column): Column = withExpr { |
There was a problem hiding this comment.
This is being merged since tz column can differ from the constant tz string for each record which makes sense in usecase, and arguably it's a common function.
|
Merged to master. |
|
@HyukjinKwon will this pr be merged into branch-2.3/branch-2.2 too? |
|
Basically, bugs are only backported to the earlier versions. |
| * @group datetime_funcs | ||
| * @since 2.4.0 | ||
| */ | ||
| def from_utc_timestamp(ts: Column, tz: Column): Column = withExpr { |
There was a problem hiding this comment.
For these changes, we also need to add the corresponding changes in PySpark functions.
|
@maropu @HyukjinKwon @zzcclp @tmnd1991 Could any of you submit a follow-up PR for PySpark? |
|
Will go ahead on this weekends. If you guys find some times please go ahead till then. |
|
ok, I get time now, so I'll take it ;) |
|
Thanks, @maropu. |
|
@gatorsmile btw, on a personal note, I got a new (& first) baby in my family hours ago ;)) |
8 participants
What changes were proposed in this pull request?
Add an overloaded version to
from_utc_timestampandto_utc_timestamphaving second argument as aColumninstead ofString.How was this patch tested?
Unit testing, especially adding two tests to org.apache.spark.sql.DateFunctionsSuite.scala