[SPARK-31256][SQL] DataFrameNaFunctions.drop should work for nested columns #28266
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
imback82
commented
Apr 19, 2020
| val exception = intercept[AnalysisException] { | ||
| df.na.drop("any", Seq("*")) | ||
| } | ||
| assert(exception.getMessage.contains("Cannot resolve column name \"*\"")) |
Contributor
Author
There was a problem hiding this comment.
Note that this was the behavior in Spark 2.4.4. We can handle this more gracefully (e.g., use outputAttributes) if we need to.
On a side note, for fill, * is ignored in Spark 2.4.4.
| val df = spark.createDataFrame( | ||
| spark.sparkContext.parallelize(data), schema) | ||
| // Nested columns are ignored for fill(). | ||
| checkAnswer(df.na.fill("a1", Seq("c1.c1-1")), df) |
Contributor
Author
There was a problem hiding this comment.
Note that nested columns are ignored for fill in Spark 2.4.4.
Contributor
Author
|
@cloud-fan Please let me know if this PR (going back to 2.4.4 behavior) makes sense. Thanks! |
imback82
changed the title
Contributor
|
nice,duplicate columns is same as struct alias which not works at toAttributes method in DataFrameNaFunctions. |
|
Test build #121490 has finished for PR 28266 at commit
|
cloud-fan
pushed a commit
that referenced
this pull request
Apr 20, 2020
…olumns ### What changes were proposed in this pull request? #26700 removed the ability to drop a row whose nested column value is null. For example, for the following `df`: ``` val schema = new StructType() .add("c1", new StructType() .add("c1-1", StringType) .add("c1-2", StringType)) val data = Seq(Row(Row(null, "a2")), Row(Row("b1", "b2")), Row(null)) val df = spark.createDataFrame(spark.sparkContext.parallelize(data), schema) df.show +--------+ | c1| +--------+ | [, a2]| |[b1, b2]| | null| +--------+ ``` In Spark 2.4.4, ``` df.na.drop("any", Seq("c1.c1-1")).show +--------+ | c1| +--------+ |[b1, b2]| +--------+ ``` In Spark 2.4.5 or Spark 3.0.0-preview2, if nested columns are specified, they are ignored. ``` df.na.drop("any", Seq("c1.c1-1")).show +--------+ | c1| +--------+ | [, a2]| |[b1, b2]| | null| +--------+ ``` ### Why are the changes needed? This seems like a regression. ### Does this PR introduce any user-facing change? Now, the nested column can be specified: ``` df.na.drop("any", Seq("c1.c1-1")).show +--------+ | c1| +--------+ |[b1, b2]| +--------+ ``` Also, if `*` is specified as a column, it will throw an `AnalysisException` that `*` cannot be resolved, which was the behavior in 2.4.4. Currently, in master, it has no effect. ### How was this patch tested? Updated existing tests. Closes #28266 from imback82/SPARK-31256. Authored-by: Terry Kim <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit d7499ae) Signed-off-by: Wenchen Fan <[email protected]>
maropu
reviewed
Apr 20, 2020
|
|
||
| checkAnswer(df.select("c1.c1-1"), | ||
| Row(null) :: Row("b1") :: Row(null) :: Nil) | ||
| test("drop with nested columns") { |
Member
There was a problem hiding this comment.
nit: This looks like a bug, so could you add the prefix: SPARK-31256.
maropu
approved these changes
Apr 20, 2020
cloud-fan
pushed a commit
that referenced
this pull request
Apr 20, 2020
…olumns
For example, for the following `df`:
```
val schema = new StructType()
.add("c1", new StructType()
.add("c1-1", StringType)
.add("c1-2", StringType))
val data = Seq(Row(Row(null, "a2")), Row(Row("b1", "b2")), Row(null))
val df = spark.createDataFrame(spark.sparkContext.parallelize(data), schema)
df.show
+--------+
| c1|
+--------+
| [, a2]|
|[b1, b2]|
| null|
+--------+
```
In Spark 2.4.4,
```
df.na.drop("any", Seq("c1.c1-1")).show
+--------+
| c1|
+--------+
|[b1, b2]|
+--------+
```
In Spark 2.4.5 or Spark 3.0.0-preview2, if nested columns are specified, they are ignored.
```
df.na.drop("any", Seq("c1.c1-1")).show
+--------+
| c1|
+--------+
| [, a2]|
|[b1, b2]|
| null|
+--------+
```
This seems like a regression.
Now, the nested column can be specified:
```
df.na.drop("any", Seq("c1.c1-1")).show
+--------+
| c1|
+--------+
|[b1, b2]|
+--------+
```
Also, if `*` is specified as a column, it will throw an `AnalysisException` that `*` cannot be resolved, which was the behavior in 2.4.4. Currently, in master, it has no effect.
Updated existing tests.
Closes #28266 from imback82/SPARK-31256.
Authored-by: Terry Kim <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit d7499ae)
Signed-off-by: Wenchen Fan <[email protected]>
Contributor
|
merging to master/3.0/2.4 |
Member
|
So, SPARK-31256 made a regression at 2.4.5 and this recovers it? |
Contributor
|
@dongjoon-hyun yes |
Member
|
Thank you for confirmation~ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
#26700 removed the ability to drop a row whose nested column value is null.
For example, for the following
df:In Spark 2.4.4,
In Spark 2.4.5 or Spark 3.0.0-preview2, if nested columns are specified, they are ignored.
Why are the changes needed?
This seems like a regression.
Does this PR introduce any user-facing change?
Now, the nested column can be specified:
Also, if
*is specified as a column, it will throw anAnalysisExceptionthat*cannot be resolved, which was the behavior in 2.4.4. Currently, in master, it has no effect.How was this patch tested?
Updated existing tests.