[SPARK-19543] from_json fails when the input row is empty

[brkyvz]

What changes were proposed in this pull request?

Using from_json on a column with an empty string results in: java.util.NoSuchElementException: head of empty list.

This is because parser.parse(input) may return Nil when input.trim.isEmpty

How was this patch tested?

Regression test in JsonExpressionsSuite

[brkyvz]

cc @hvanhovell mind taking a look? You looked at the last PR that touched this code

[HyukjinKwon]

(Not for this PR but maybe loosely related I guess) I was thinking it is a bit odd that we support to only read the single row when it is a json array. It seems, for example,

import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
val schema = StructType(StructField("a", IntegerType) :: Nil)
Seq(("""[{"a": 1}, {"a": 2}]""")).toDF("struct").select(from_json(col("struct"), schema)).show()
+--------------------+
|jsontostruct(struct)|
+--------------------+
|                 [1]|
+--------------------+

I think maybe we should not support this in that function or it should work like a generator expression.

[hvanhovell]

@HyukjinKwon that seems fair. Feel free to work in this.

[HyukjinKwon]

Thank you for confirming.

[SparkQA]

Test build #72681 has finished for PR 16881 at commit 6ef0f45.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[hvanhovell]

LGTM - merging to master/2.1. Thanks!

[yeikel]

Is there any way to use this fix without re-installing spark? Or is there any workaround?

In my organization they are running a version of Spark where this error is happening but there are no plans to upgrade anytime soon.

[HyukjinKwon]

You can manually replace the empty strings to, for instance, empty object {} by dataframe APIs before call from_json.

[yeikel]

@HyukjinKwon I've tried your suggestion with the following attempt :

val empty = df.na.fill("{}",Seq("phone"))
val json_columns = df.withColumn("phone", from_json($"phone", schema))
json_columns.show

But it still gave me the same error. Any suggestions?

Thank you

[HyukjinKwon]

Can you show your df? Btw let's ask a question to mailing list next time. Here usually is supposed to discuss about the change itself.

[yeikel]

@HyukjinKwon I fixed it. Problem was the string '[]' replacing it with '{}' fixed the issue . I am not sure if the function should be able to parse '[]' tho.

I commented here because I thought devs could give a better input, but I will post in the email list next time.

Thank you