[SPARK-24640][SQL] Return NULL from size(NULL) by default
#26051
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
@HyukjinKwon @cloud-fan @gatorsmile Please, take a look at the PR. |
cloud-fan
approved these changes
Oct 8, 2019
|
Test build #111880 has finished for PR 26051 at commit
|
HyukjinKwon
approved these changes
Oct 8, 2019
Member
|
Merged to master. |
Member
|
@MaxGekk Could you submit a follow-up PR to update the migration guide? |
Member
Author
|
Here is the PR #26066 which updates the SQL migration guide. |
atronchi
pushed a commit
to atronchi/spark
that referenced
this pull request
Oct 23, 2019
### What changes were proposed in this pull request? Set the default value of the `spark.sql.legacy.sizeOfNull` config to `false`. That changes behavior of the `size()` function for `NULL`. The function will return `NULL` for `NULL` instead of `-1`. ### Why are the changes needed? There is the agreement in the PR apache#21598 (comment) to change behavior in Spark 3.0. ### Does this PR introduce any user-facing change? Yes. Before: ```sql spark-sql> select size(NULL); -1 ``` After: ```sql spark-sql> select size(NULL); NULL ``` ### How was this patch tested? By the `check outputs of expression examples` test in `SQLQuerySuite` which runs expression examples. Closes apache#26051 from MaxGekk/sizeof-null-returns-null. Authored-by: Maxim Gekk <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
dongjoon-hyun
pushed a commit
that referenced
this pull request
Mar 11, 2020
…efault ### What changes were proposed in this pull request? This PR reverts #26051 and #26066 ### Why are the changes needed? There is no standard requiring that `size(null)` must return null, and returning -1 looks reasonable as well. This is kind of a cosmetic change and we should avoid it if it breaks existing queries. This is similar to reverting TRIM function parameter order change. ### Does this PR introduce any user-facing change? Yes, change the behavior of `size(null)` back to be the same as 2.4. ### How was this patch tested? N/A Closes #27834 from cloud-fan/revert. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
dongjoon-hyun
pushed a commit
that referenced
this pull request
Mar 11, 2020
…efault ### What changes were proposed in this pull request? This PR reverts #26051 and #26066 ### Why are the changes needed? There is no standard requiring that `size(null)` must return null, and returning -1 looks reasonable as well. This is kind of a cosmetic change and we should avoid it if it breaks existing queries. This is similar to reverting TRIM function parameter order change. ### Does this PR introduce any user-facing change? Yes, change the behavior of `size(null)` back to be the same as 2.4. ### How was this patch tested? N/A Closes #27834 from cloud-fan/revert. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 8efb710) Signed-off-by: Dongjoon Hyun <[email protected]>
sjincho
pushed a commit
to sjincho/spark
that referenced
this pull request
Apr 15, 2020
…efault ### What changes were proposed in this pull request? This PR reverts apache#26051 and apache#26066 ### Why are the changes needed? There is no standard requiring that `size(null)` must return null, and returning -1 looks reasonable as well. This is kind of a cosmetic change and we should avoid it if it breaks existing queries. This is similar to reverting TRIM function parameter order change. ### Does this PR introduce any user-facing change? Yes, change the behavior of `size(null)` back to be the same as 2.4. ### How was this patch tested? N/A Closes apache#27834 from cloud-fan/revert. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Set the default value of the
spark.sql.legacy.sizeOfNullconfig tofalse. That changes behavior of thesize()function forNULL. The function will returnNULLforNULLinstead of-1.Why are the changes needed?
There is the agreement in the PR #21598 (comment) to change behavior in Spark 3.0.
Does this PR introduce any user-facing change?
Yes.
Before:
After:
How was this patch tested?
By the
check outputs of expression examplestest inSQLQuerySuitewhich runs expression examples.