[SPARK-40292][SQL] Fix column names in "arrays_zip" function when arrays are referenced from nested structs #37833
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sadikovi
changed the title
sadikovi
commented
Sep 8, 2022
|
|
||
| val fieldNames = res.schema.head.dataType.asInstanceOf[ArrayType] | ||
| .elementType.asInstanceOf[StructType].fieldNames | ||
| assert(fieldNames.toSeq === Seq("arr_1", "arr_2", "arr_3")) |
Contributor
Author
There was a problem hiding this comment.
Before the fix, this would return Seq("0", "1", "2").
|
Can one of the admins verify this patch? |
MaxGekk
approved these changes
Sep 10, 2022
Member
@sadikovi To where should we backport this? Up to branch-3.1 inclusively? |
Contributor
Author
|
Thanks @MaxGekk. Spark 3.1 produces the correct result so I think we can just backport to Spark 3.2 and Spark 3.3. |
Member
|
+1, LGTM. Merging to master/3.3/3.2. |
MaxGekk
pushed a commit
that referenced
this pull request
Sep 12, 2022
…ays are referenced from nested structs
### What changes were proposed in this pull request?
This PR fixes an issue in `arrays_zip` function where a field index was used as a column name in the resulting schema which was a regression from Spark 3.1. With this change, the original behaviour is restored: a corresponding struct field name will be used instead of a field index.
Example:
```sql
with q as (
select
named_struct(
'my_array', array(1, 2, 3),
'my_array2', array(4, 5, 6)
) as my_struct
)
select
arrays_zip(my_struct.my_array, my_struct.my_array2)
from
q
```
would return schema:
```
root
|-- arrays_zip(my_struct.my_array, my_struct.my_array2): array (nullable = false)
| |-- element: struct (containsNull = false)
| | |-- 0: integer (nullable = true)
| | |-- 1: integer (nullable = true)
```
which is somewhat inaccurate. PR adds handling of `GetStructField` expression to return the struct field names like this:
```
root
|-- arrays_zip(my_struct.my_array, my_struct.my_array2): array (nullable = false)
| |-- element: struct (containsNull = false)
| | |-- my_array: integer (nullable = true)
| | |-- my_array2: integer (nullable = true)
```
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
Yes, `arrays_zip` function returns struct field names now as in Spark 3.1 instead of field indices.
Some users might have worked around this issue so this patch would affect them by bringing back the original behaviour.
### How was this patch tested?
Existing unit tests. I also added a test case that reproduces the problem.
Closes #37833 from sadikovi/SPARK-40292.
Authored-by: Ivan Sadikov <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
(cherry picked from commit 443eea9)
Signed-off-by: Max Gekk <[email protected]>
MaxGekk
pushed a commit
that referenced
this pull request
Sep 12, 2022
…ays are referenced from nested structs
### What changes were proposed in this pull request?
This PR fixes an issue in `arrays_zip` function where a field index was used as a column name in the resulting schema which was a regression from Spark 3.1. With this change, the original behaviour is restored: a corresponding struct field name will be used instead of a field index.
Example:
```sql
with q as (
select
named_struct(
'my_array', array(1, 2, 3),
'my_array2', array(4, 5, 6)
) as my_struct
)
select
arrays_zip(my_struct.my_array, my_struct.my_array2)
from
q
```
would return schema:
```
root
|-- arrays_zip(my_struct.my_array, my_struct.my_array2): array (nullable = false)
| |-- element: struct (containsNull = false)
| | |-- 0: integer (nullable = true)
| | |-- 1: integer (nullable = true)
```
which is somewhat inaccurate. PR adds handling of `GetStructField` expression to return the struct field names like this:
```
root
|-- arrays_zip(my_struct.my_array, my_struct.my_array2): array (nullable = false)
| |-- element: struct (containsNull = false)
| | |-- my_array: integer (nullable = true)
| | |-- my_array2: integer (nullable = true)
```
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
Yes, `arrays_zip` function returns struct field names now as in Spark 3.1 instead of field indices.
Some users might have worked around this issue so this patch would affect them by bringing back the original behaviour.
### How was this patch tested?
Existing unit tests. I also added a test case that reproduces the problem.
Closes #37833 from sadikovi/SPARK-40292.
Authored-by: Ivan Sadikov <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
(cherry picked from commit 443eea9)
Signed-off-by: Max Gekk <[email protected]>
HyukjinKwon
pushed a commit
that referenced
this pull request
Sep 16, 2022
…rays_zip" function ### What changes were proposed in this pull request? This is a follow-up for #37833. The PR fixes column names in `arrays_zip` function for the cases when `GetArrayStructFields` and `GetMapValue` expressions are used (see unit tests for more details). Before the patch, the column names would be indexes or an AnalysisException would be thrown in the case of `GetArrayStructFields` example. ### Why are the changes needed? Fixes an inconsistency issue in Spark 3.2 and onwards where the fields would be labeled as indexes instead of column names. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I added unit tests that reproduce the issue and confirmed that the patch fixes them. Closes #37911 from sadikovi/SPARK-40470. Authored-by: Ivan Sadikov <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
HyukjinKwon
pushed a commit
that referenced
this pull request
Sep 16, 2022
…rays_zip" function ### What changes were proposed in this pull request? This is a follow-up for #37833. The PR fixes column names in `arrays_zip` function for the cases when `GetArrayStructFields` and `GetMapValue` expressions are used (see unit tests for more details). Before the patch, the column names would be indexes or an AnalysisException would be thrown in the case of `GetArrayStructFields` example. ### Why are the changes needed? Fixes an inconsistency issue in Spark 3.2 and onwards where the fields would be labeled as indexes instead of column names. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I added unit tests that reproduce the issue and confirmed that the patch fixes them. Closes #37911 from sadikovi/SPARK-40470. Authored-by: Ivan Sadikov <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]> (cherry picked from commit 9b0f979) Signed-off-by: Hyukjin Kwon <[email protected]>
HyukjinKwon
pushed a commit
that referenced
this pull request
Sep 16, 2022
…rays_zip" function ### What changes were proposed in this pull request? This is a follow-up for #37833. The PR fixes column names in `arrays_zip` function for the cases when `GetArrayStructFields` and `GetMapValue` expressions are used (see unit tests for more details). Before the patch, the column names would be indexes or an AnalysisException would be thrown in the case of `GetArrayStructFields` example. ### Why are the changes needed? Fixes an inconsistency issue in Spark 3.2 and onwards where the fields would be labeled as indexes instead of column names. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I added unit tests that reproduce the issue and confirmed that the patch fixes them. Closes #37911 from sadikovi/SPARK-40470. Authored-by: Ivan Sadikov <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]> (cherry picked from commit 9b0f979) Signed-off-by: Hyukjin Kwon <[email protected]>
LuciferYang
pushed a commit
to LuciferYang/spark
that referenced
this pull request
Sep 20, 2022
…rays_zip" function ### What changes were proposed in this pull request? This is a follow-up for apache#37833. The PR fixes column names in `arrays_zip` function for the cases when `GetArrayStructFields` and `GetMapValue` expressions are used (see unit tests for more details). Before the patch, the column names would be indexes or an AnalysisException would be thrown in the case of `GetArrayStructFields` example. ### Why are the changes needed? Fixes an inconsistency issue in Spark 3.2 and onwards where the fields would be labeled as indexes instead of column names. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I added unit tests that reproduce the issue and confirmed that the patch fixes them. Closes apache#37911 from sadikovi/SPARK-40470. Authored-by: Ivan Sadikov <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
sunchao
pushed a commit
to sunchao/spark
that referenced
this pull request
Jun 2, 2023
…ays are referenced from nested structs
### What changes were proposed in this pull request?
This PR fixes an issue in `arrays_zip` function where a field index was used as a column name in the resulting schema which was a regression from Spark 3.1. With this change, the original behaviour is restored: a corresponding struct field name will be used instead of a field index.
Example:
```sql
with q as (
select
named_struct(
'my_array', array(1, 2, 3),
'my_array2', array(4, 5, 6)
) as my_struct
)
select
arrays_zip(my_struct.my_array, my_struct.my_array2)
from
q
```
would return schema:
```
root
|-- arrays_zip(my_struct.my_array, my_struct.my_array2): array (nullable = false)
| |-- element: struct (containsNull = false)
| | |-- 0: integer (nullable = true)
| | |-- 1: integer (nullable = true)
```
which is somewhat inaccurate. PR adds handling of `GetStructField` expression to return the struct field names like this:
```
root
|-- arrays_zip(my_struct.my_array, my_struct.my_array2): array (nullable = false)
| |-- element: struct (containsNull = false)
| | |-- my_array: integer (nullable = true)
| | |-- my_array2: integer (nullable = true)
```
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
Yes, `arrays_zip` function returns struct field names now as in Spark 3.1 instead of field indices.
Some users might have worked around this issue so this patch would affect them by bringing back the original behaviour.
### How was this patch tested?
Existing unit tests. I also added a test case that reproduces the problem.
Closes apache#37833 from sadikovi/SPARK-40292.
Authored-by: Ivan Sadikov <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
(cherry picked from commit 443eea9)
Signed-off-by: Max Gekk <[email protected]>
sunchao
pushed a commit
to sunchao/spark
that referenced
this pull request
Jun 2, 2023
…rays_zip" function ### What changes were proposed in this pull request? This is a follow-up for apache#37833. The PR fixes column names in `arrays_zip` function for the cases when `GetArrayStructFields` and `GetMapValue` expressions are used (see unit tests for more details). Before the patch, the column names would be indexes or an AnalysisException would be thrown in the case of `GetArrayStructFields` example. ### Why are the changes needed? Fixes an inconsistency issue in Spark 3.2 and onwards where the fields would be labeled as indexes instead of column names. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I added unit tests that reproduce the issue and confirmed that the patch fixes them. Closes apache#37911 from sadikovi/SPARK-40470. Authored-by: Ivan Sadikov <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]> (cherry picked from commit 9b0f979) Signed-off-by: Hyukjin Kwon <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR fixes an issue in
arrays_zipfunction where a field index was used as a column name in the resulting schema which was a regression from Spark 3.1. With this change, the original behaviour is restored: a corresponding struct field name will be used instead of a field index.Example:
would return schema:
which is somewhat inaccurate. PR adds handling of
GetStructFieldexpression to return the struct field names like this:Why are the changes needed?
Does this PR introduce any user-facing change?
Yes,
arrays_zipfunction returns struct field names now as in Spark 3.1 instead of field indices.Some users might have worked around this issue so this patch would affect them by bringing back the original behaviour.
How was this patch tested?
Existing unit tests. I also added a test case that reproduces the problem.