Skip to content

Conversation

@jasoncl
Copy link
Contributor

@jasoncl jasoncl commented Oct 2, 2015

The error message is now changed from "Do not support type class scala.Tuple2." to "Do not support type class org.json4s.JsonAST$JNull$" to be more informative about what is not supported. Also, StructType metadata now handles JNull correctly, i.e., {'a': None}. test_metadata_null is added to tests.py to show the fix works.

@shea-parkes
Copy link
Contributor

Thanks for doing this!

@JoshRosen
Copy link
Contributor

Jenkins, this is ok to test.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is incorrect. Also, please leave an additional line of whitespace before this line.

@JoshRosen
Copy link
Contributor

This seems good overall, but one high-level question: why store None on the Java side instead of null? I'm just wondering whether mapping it back to Java null would make more sense. I guess it doesn't make a difference from Python's point of view, since Python's view of the metadata is preserved roundtrip, but it could hypothetically matter if Python is trying to set metadata which is then consumed by Java library code (for example).

@SparkQA
Copy link

SparkQA commented Oct 18, 2015

Test build #43900 has finished for PR 8969 at commit 6bb4fcc.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 23, 2015

Test build #44246 has finished for PR 8969 at commit 5524a92.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 23, 2015

Test build #44243 has finished for PR 8969 at commit 978fdc7.

  • This patch passes all tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@jasoncl
Copy link
Contributor Author

jasoncl commented Oct 23, 2015

The test build #44243 for commit 978fdc7 shows that the patch does not merge cleanly. But that is not my latest commit. My latest commit 5524a92 has passed the test build and will merge cleanly.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like not worth adding a method since it is just used once?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The method is created for code organization and clarity purpose. This way it can be easily reused in the future.

@yhuai
Copy link
Contributor

yhuai commented Jan 13, 2016

@jasoncl Can you do a quick update if my comment makes sense? Then, we will get it merged. Thanks!

@JoshRosen
Copy link
Contributor

Ping @yhuai, is this ready for merging?

@JoshRosen
Copy link
Contributor

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Jan 27, 2016

Test build #50160 has finished for PR 8969 at commit 5524a92.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yhuai
Copy link
Contributor

yhuai commented Jan 27, 2016

LGTM. I am merging it.

asfgit pushed a commit that referenced this pull request Jan 27, 2016
…ith `None` triggers cryptic failure

The error message is now changed from "Do not support type class scala.Tuple2." to "Do not support type class org.json4s.JsonAST$JNull$" to be more informative about what is not supported. Also, StructType metadata now handles JNull correctly, i.e., {'a': None}. test_metadata_null is added to tests.py to show the fix works.

Author: Jason Lee <[email protected]>

Closes #8969 from jasoncl/SPARK-10847.

(cherry picked from commit edd4737)
Signed-off-by: Yin Huai <[email protected]>
asfgit pushed a commit that referenced this pull request Jan 27, 2016
…ith `None` triggers cryptic failure

The error message is now changed from "Do not support type class scala.Tuple2." to "Do not support type class org.json4s.JsonAST$JNull$" to be more informative about what is not supported. Also, StructType metadata now handles JNull correctly, i.e., {'a': None}. test_metadata_null is added to tests.py to show the fix works.

Author: Jason Lee <[email protected]>

Closes #8969 from jasoncl/SPARK-10847.

(cherry picked from commit edd4737)
Signed-off-by: Yin Huai <[email protected]>
@asfgit asfgit closed this in edd4737 Jan 27, 2016
@yhuai
Copy link
Contributor

yhuai commented Jan 27, 2016

I also merged it in branch 1.5 and branch 1.6 since it is very isolated bug fix.

@shea-parkes
Copy link
Contributor

Thanks a bunch guys, this will be a big help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants