Skip to content

Commit 87f28fc

Browse files
cloud-fanmarmbrus
authored andcommitted
[SPARK-11379][SQL] ExpressionEncoder can't handle top level primitive type correctly
For inner primitive type(e.g. inside `Product`), we use `schemaFor` to get the catalyst type for it, https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala#L403. However, for top level primitive type, we use `dataTypeFor`, which is wrong. Author: Wenchen Fan <[email protected]> Closes #9337 from cloud-fan/encoder.
1 parent 3dfa4ea commit 87f28fc

File tree

2 files changed

+2
-1
lines changed

2 files changed

+2
-1
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -170,7 +170,7 @@ trait ScalaReflection {
170170
.getOrElse(BoundReference(ordinal, dataType, false))
171171

172172
/** Returns the current path or throws an error. */
173-
def getPath = path.getOrElse(BoundReference(0, dataTypeFor(tpe), true))
173+
def getPath = path.getOrElse(BoundReference(0, schemaFor(tpe).dataType, true))
174174

175175
tpe match {
176176
case t if !dataTypeFor(t).isInstanceOf[ObjectType] =>

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,7 @@ class ExpressionEncoderSuite extends SparkFunSuite {
5757
encodeDecodeTest(false)
5858
encodeDecodeTest(1.toShort)
5959
encodeDecodeTest(1.toByte)
60+
encodeDecodeTest("hello")
6061

6162
encodeDecodeTest(PrimitiveData(1, 1, 1, 1, 1, 1, true))
6263

0 commit comments

Comments
 (0)