[SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type

[HyukjinKwon]

What changes were proposed in this pull request?

This PR targets for non-nullable null type not to coerce to nullable type in complex types.

Non-nullable fields in struct, elements in an array and entries in map can mean empty array, struct and map. They are empty so it does not need to force the nullability when we find common types.

This PR also reverts and supersedes d7b97a1

Why are the changes needed?

To make type coercion coherent and consistent. Currently, we correctly keep the nullability even between non-nullable fields:

import org.apache.spark.sql.types._
import org.apache.spark.sql.functions._
spark.range(1).select(array(lit(1)).cast(ArrayType(IntegerType, false))).printSchema()
spark.range(1).select(array(lit(1)).cast(ArrayType(DoubleType, false))).printSchema()

spark.range(1).selectExpr("concat(array(1), array(1)) as arr").printSchema()

Does this PR introduce any user-facing change?

Yes.

import org.apache.spark.sql.types._
import org.apache.spark.sql.functions._
spark.range(1).select(array().cast(ArrayType(IntegerType, false))).printSchema()

spark.range(1).selectExpr("concat(array(), array(1)) as arr").printSchema()

Before:

org.apache.spark.sql.AnalysisException: cannot resolve 'array()' due to data type mismatch: cannot cast array<null> to array<int>;;
'Project [cast(array() as array<int>) AS array()#68]
+- Range (0, 1, step=1, splits=Some(12))

  at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:149)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:140)
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$2(TreeNode.scala:333)
  at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:72)
  at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:333)
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$1(TreeNode.scala:330)
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:399)
  at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:237)

root
 |-- arr: array (nullable = false)
 |    |-- element: integer (containsNull = true)

After:

root
 |-- array(): array (nullable = false)
 |    |-- element: integer (containsNull = false)

root
 |-- arr: array (nullable = false)
 |    |-- element: integer (containsNull = false)

How was this patch tested?

Unittests were added and manually tested.

[SparkQA]

Test build #120206 has finished for PR 27991 at commit 7c54a5f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #120209 has finished for PR 27991 at commit 51acfeb.

• This patch fails PySpark pip packaging tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #120226 has finished for PR 27991 at commit dfd9343.

• This patch fails PySpark pip packaging tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #120246 has finished for PR 27991 at commit 7bae2c1.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #120235 has finished for PR 27991 at commit e4557ae.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

retest this please

[SparkQA]

Test build #120261 has finished for PR 27991 at commit 7bae2c1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #120292 has finished for PR 27991 at commit 05dc916.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #120356 has finished for PR 27991 at commit d57d4a6.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #120360 has finished for PR 27991 at commit 1bf68f0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to master/3.0!

[bart-samwel]

How about concat(array(), array(NULL))? That should have the same type as array(NULL).

[maropu]

Yea, the output has the same type with array(null);

scala> sql("select concat(array(), array(NULL))").printSchema
root
 |-- concat(array(), array(NULL)): array (nullable = false)
 |    |-- element: null (containsNull = true)

scala> sql("select array()").printSchema
root
 |-- array(): array (nullable = false)
 |    |-- element: null (containsNull = false)

scala> sql("select array(null)").printSchema
root
 |-- array(NULL): array (nullable = false)
 |    |-- element: null (containsNull = true)

Any concern?

[bart-samwel]

Only that it should be tested, since it's an interesting corner case!

[HyukjinKwon]

I believe those cases are tested in CastSuite.scala and TypeCoercionSuite.scala including all types if I didn't miss anything. I just kept one e2e test here since it was the reported case in the JIRA SPARK-31227.