Skip to content

Conversation

@cloud-fan
Copy link
Contributor

nullability should only be considered as an optimization rather than part of the type system, so instead of failing analysis for mismatch nullability, we should pass analysis and add runtime null check.

backport #11035 to 1.6

@SparkQA
Copy link

SparkQA commented Feb 3, 2016

Test build #50635 has finished for PR 11042 at commit c2bc287.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Feb 3, 2016

Test build #50639 has finished for PR 11042 at commit 9f8c41e.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan cloud-fan force-pushed the branch-1.6 branch 2 times, most recently from 9845bed to c3f659b Compare February 3, 2016 05:53
@cloud-fan
Copy link
Contributor Author

The error message looks like:

[info]   Cause: java.lang.RuntimeException: Null value appeared in non-nullable field:
[info] - field (class: "scala.Int", name: "b")
[info] - field (class: "org.apache.spark.sql.ClassData", name: "f")
[info] - root class: "org.apache.spark.sql.NestedStruct"
[info] If the schema is inferred from a Scala tuple/case class, or a Java bean, please try to use scala.Option[_] or other nullable types (e.g. java.lang.Integer instead of int/scala.Int).

@liancheng
Copy link
Contributor

Memo: CodegenContext.references can only hold Expression objects in branch-1.6. That's why the error message can't be moved to codegen references like what #11035 does.

@SparkQA
Copy link

SparkQA commented Feb 3, 2016

Test build #50648 has finished for PR 11042 at commit 6b5a100.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@liancheng
Copy link
Contributor

LGTM

@marmbrus
Copy link
Contributor

marmbrus commented Feb 4, 2016

Thanks, merging.

asfgit pushed a commit that referenced this pull request Feb 4, 2016
…ld not fail analysis of encoder

nullability should only be considered as an optimization rather than part of the type system, so instead of failing analysis for mismatch nullability, we should pass analysis and add runtime null check.

backport #11035 to 1.6

Author: Wenchen Fan <[email protected]>

Closes #11042 from cloud-fan/branch-1.6.
@cloud-fan cloud-fan closed this Feb 5, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants