[SPARK-13101][SQL] nullability of array type element should not fail analysis of encoder

[cloud-fan]

nullability should only be considered as an optimization rather than part of the type system, so instead of failing analysis for mismatch nullability, we should pass analysis and add runtime null check.

[cloud-fan]

cc @marmbrus @liancheng

[cloud-fan]

The previous tests are very verbose and hard to maintain, every time we change some unrelated analysis behaviours, these tests need to be updated. Actually for these tests, we only care about if analysis passes, runtime execution successes. So I simplified these tests to only ensure we don't fail analysis and execution.

[marmbrus]

+1

[SparkQA]

Test build #50611 has finished for PR 11035 at commit aa07e56.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class AssertNotNull(child: Expression, walkedTypePath: Seq[String])

[liancheng]

retest this please

[SparkQA]

Test build #50617 has finished for PR 11035 at commit aa07e56.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class AssertNotNull(child: Expression, walkedTypePath: Seq[String])

[cloud-fan]

retest this please

[liancheng]

I'm investigating the failed flaky tests, and @davies is marking them as ignored temporarily to bring back the PR builder.

[SparkQA]

Test build #50620 has finished for PR 11035 at commit aa07e56.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class AssertNotNull(child: Expression, walkedTypePath: Seq[String])

[cloud-fan]

retest it please

[liancheng]

LGTM

[marmbrus]

So this does still silently corrupt values.

scala> Seq(("a", Seq(null, new Integer(1)))).toDS().as[(String, Array[Int])].collect()
res5: Array[(String, Array[Int])] = Array((a,Array(0, 1)))

Since this isn't a regression its probably okay as long as we open another JIRA.

[cloud-fan]

added https://issues.apache.org/jira/browse/SPARK-13155

[SparkQA]

Test build #2507 has finished for PR 11035 at commit aa07e56.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class AssertNotNull(child: Expression, walkedTypePath: Seq[String])

[cloud-fan]

retest this please

[cloud-fan]

retest this please

[SparkQA]

Test build #50860 has finished for PR 11035 at commit aa07e56.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class AssertNotNull(child: Expression, walkedTypePath: Seq[String])

[cloud-fan]

retest this please

[SparkQA]

Test build #50861 has finished for PR 11035 at commit aa07e56.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class AssertNotNull(child: Expression, walkedTypePath: Seq[String])

[cloud-fan]

retest this please

[SparkQA]

Test build #50864 has finished for PR 11035 at commit aa07e56.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class AssertNotNull(child: Expression, walkedTypePath: Seq[String])

[gatorsmile]

retest this please

[SparkQA]

Test build #50871 has finished for PR 11035 at commit aa07e56.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class AssertNotNull(child: Expression, walkedTypePath: Seq[String])

[cloud-fan]

retest this please

[SparkQA]

Test build #50886 has finished for PR 11035 at commit aa07e56.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class AssertNotNull(child: Expression, walkedTypePath: Seq[String])

[cloud-fan]

cc @marmbrus

[marmbrus]

Thanks, merging to master.