-
Notifications
You must be signed in to change notification settings - Fork 28.9k
[SPARK-10847] [SQL] [PySpark] Pyspark - DataFrame - Optional Metadata with None triggers cryptic failure
#8969
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…ctual unsupported type in the tuple instead of just tuple2
|
Thanks for doing this! |
|
Jenkins, this is ok to test. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This comment is incorrect. Also, please leave an additional line of whitespace before this line.
|
This seems good overall, but one high-level question: why store |
|
Test build #43900 has finished for PR 8969 at commit
|
|
Test build #44246 has finished for PR 8969 at commit
|
|
Test build #44243 has finished for PR 8969 at commit
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like not worth adding a method since it is just used once?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The method is created for code organization and clarity purpose. This way it can be easily reused in the future.
|
@jasoncl Can you do a quick update if my comment makes sense? Then, we will get it merged. Thanks! |
|
Ping @yhuai, is this ready for merging? |
|
Jenkins, retest this please. |
|
Test build #50160 has finished for PR 8969 at commit
|
|
LGTM. I am merging it. |
…ith `None` triggers cryptic failure
The error message is now changed from "Do not support type class scala.Tuple2." to "Do not support type class org.json4s.JsonAST$JNull$" to be more informative about what is not supported. Also, StructType metadata now handles JNull correctly, i.e., {'a': None}. test_metadata_null is added to tests.py to show the fix works.
Author: Jason Lee <[email protected]>
Closes #8969 from jasoncl/SPARK-10847.
(cherry picked from commit edd4737)
Signed-off-by: Yin Huai <[email protected]>
…ith `None` triggers cryptic failure
The error message is now changed from "Do not support type class scala.Tuple2." to "Do not support type class org.json4s.JsonAST$JNull$" to be more informative about what is not supported. Also, StructType metadata now handles JNull correctly, i.e., {'a': None}. test_metadata_null is added to tests.py to show the fix works.
Author: Jason Lee <[email protected]>
Closes #8969 from jasoncl/SPARK-10847.
(cherry picked from commit edd4737)
Signed-off-by: Yin Huai <[email protected]>
|
I also merged it in branch 1.5 and branch 1.6 since it is very isolated bug fix. |
|
Thanks a bunch guys, this will be a big help. |
The error message is now changed from "Do not support type class scala.Tuple2." to "Do not support type class org.json4s.JsonAST$JNull$" to be more informative about what is not supported. Also, StructType metadata now handles JNull correctly, i.e., {'a': None}. test_metadata_null is added to tests.py to show the fix works.