[SPARK-36567][SQL] Support foldable special datetime strings by CAST
#33816
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #142708 has finished for PR 33816 at commit
|
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Test build #142709 has finished for PR 33816 at commit
|
HyukjinKwon
reviewed
Aug 24, 2021
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala
Show resolved
Hide resolved
|
Kubernetes integration test starting |
MaxGekk
changed the title
CASTCAST
|
Kubernetes integration test status failure |
|
@cloud-fan @gengliangwang @HyukjinKwon Could you review this PR, please. |
|
Test build #142741 has finished for PR 33816 at commit
|
cloud-fan
approved these changes
Aug 25, 2021
|
thanks, merging to master/3.2! |
cloud-fan
pushed a commit
that referenced
this pull request
Aug 25, 2021
### What changes were proposed in this pull request? In the PR, I propose to add new correctness rule `SpecialDatetimeValues` to the final analysis phase. It replaces casts of strings to date/timestamp_ltz/timestamp_ntz by literals of such types if the strings contain special datetime values like `today`, `yesterday` and `tomorrow`, and the input strings are foldable. ### Why are the changes needed? 1. To avoid a breaking change. 2. To improve user experience with Spark SQL. After the PR #32714, users have to use typed literals instead of implicit casts. For instance, at Spark 3.1: ```sql select ts_col > 'now'; ``` but the query fails at the moment, and users have to use typed timestamp literal: ```sql select ts_col > timestamp'now'; ``` ### Does this PR introduce _any_ user-facing change? No. Previous release 3.1 has supported the feature already till it was removed by #32714. ### How was this patch tested? 1. Manually tested via the sql command line: ```sql spark-sql> select cast('today' as date); 2021-08-24 spark-sql> select timestamp('today'); 2021-08-24 00:00:00 spark-sql> select timestamp'tomorrow' > 'today'; true ``` 2. By running new test suite: ``` $ build/sbt "sql/testOnly org.apache.spark.sql.catalyst.optimizer.SpecialDatetimeValuesSuite" ``` Closes #33816 from MaxGekk/foldable-datetime-special-values. Authored-by: Max Gekk <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit df0ec56) Signed-off-by: Wenchen Fan <[email protected]>
|
@MaxGekk I think we should also update the migration guide? |
|
Here is the update for the SQL migration guide #33840 |
cloud-fan
pushed a commit
that referenced
this pull request
Aug 26, 2021
…ion guide about foldable special datetime values ### What changes were proposed in this pull request? In the PR, I propose to update an existing item in the SQL migration guide, and mention that Spark 3.2 supports foldable special datetime values as well. <img width="1292" alt="Screenshot 2021-08-25 at 23 29 51" src="https://user-images.githubusercontent.com/1580697/130860184-27f0ba56-6c2d-4a5a-91a8-195f2f8aa5da.png"> ### Why are the changes needed? To inform users about actual Spark SQL behavior introduced by #33816 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? By generating docs, and checking results manually. Closes #33840 from MaxGekk/special-datetime-cast-migr-guide. Authored-by: Max Gekk <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
cloud-fan
pushed a commit
that referenced
this pull request
Aug 26, 2021
…ion guide about foldable special datetime values ### What changes were proposed in this pull request? In the PR, I propose to update an existing item in the SQL migration guide, and mention that Spark 3.2 supports foldable special datetime values as well. <img width="1292" alt="Screenshot 2021-08-25 at 23 29 51" src="https://user-images.githubusercontent.com/1580697/130860184-27f0ba56-6c2d-4a5a-91a8-195f2f8aa5da.png"> ### Why are the changes needed? To inform users about actual Spark SQL behavior introduced by #33816 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? By generating docs, and checking results manually. Closes #33840 from MaxGekk/special-datetime-cast-migr-guide. Authored-by: Max Gekk <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit c4e739f) Signed-off-by: Wenchen Fan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
In the PR, I propose to add new correctness rule
SpecialDatetimeValuesto the final analysis phase. It replaces casts of strings to date/timestamp_ltz/timestamp_ntz by literals of such types if the strings contain special datetime values liketoday,yesterdayandtomorrow, and the input strings are foldable.Why are the changes needed?
at Spark 3.1:
but the query fails at the moment, and users have to use typed timestamp literal:
Does this PR introduce any user-facing change?
No. Previous release 3.1 has supported the feature already till it was removed by #32714.
How was this patch tested?