Skip to content

[SPARK-18398][SQL] Fix nullabilities of MapObjects and ExternalMapToCatalyst. #15840

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 11 commits into from
Closed

[SPARK-18398][SQL] Fix nullabilities of MapObjects and ExternalMapToCatalyst. #15840

Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
b287ded
Fix nullabilities of MapObjects and optimize not to check null if lam…
ueshin Nov 10, 2016
deffccd
Use `ctx.nullSafeExec()` to generate nullability checking in `MapObje…
ueshin Nov 11, 2016
0fc36a8
Use `ctx.nullSafeExec()` to generate nullability checking.
ueshin Nov 14, 2016
658c5e3
Fix nullability of map values of `ExternalMapToCatalyst`.
ueshin Nov 14, 2016
ec0c55c
Use `ctx.nullSafeExec()` to generate nullability checking in complex …
ueshin Nov 14, 2016
727f5a9
Revert "Use `ctx.nullSafeExec()` to generate nullability checking in …
ueshin Nov 21, 2016
dee450e
Revert "Use `ctx.nullSafeExec()` to generate nullability checking."
ueshin Nov 21, 2016
cad3d74
Revert "Use `ctx.nullSafeExec()` to generate nullability checking in …
ueshin Nov 21, 2016
560152b
Merge branch 'master' into issues/SPARK-18398
ueshin Nov 21, 2016
78d9e80
Modify generated codes to be optimized by Janino.
ueshin Nov 21, 2016
e777fbc
Revert replacements.
ueshin Nov 21, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -461,14 +461,15 @@ case class MapObjects private(
lambdaFunction: Expression,
inputData: Expression) extends Expression with NonSQLExpression {

override def nullable: Boolean = true
override def nullable: Boolean = inputData.nullable

override def children: Seq[Expression] = lambdaFunction :: inputData :: Nil

override def eval(input: InternalRow): Any =
throw new UnsupportedOperationException("Only code-generated evaluation is supported")

override def dataType: DataType = ArrayType(lambdaFunction.dataType)
override def dataType: DataType =
ArrayType(lambdaFunction.dataType, containsNull = lambdaFunction.nullable)

override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val elementJavaType = ctx.javaType(loopVarDataType)
Expand Down Expand Up @@ -642,7 +643,8 @@ case class ExternalMapToCatalyst private(

override def foldable: Boolean = false

override def dataType: MapType = MapType(keyConverter.dataType, valueConverter.dataType)
override def dataType: MapType = MapType(
keyConverter.dataType, valueConverter.dataType, valueContainsNull = valueConverter.nullable)

override def eval(input: InternalRow): Any =
throw new UnsupportedOperationException("Only code-generated evaluation is supported")
Expand Down