[SPARK-21513][SQL][FOLLOWUP] Allow UDF to_json support converting MapType to json for PySpark and SparkR #19223

goldmedal · 2017-09-13T19:55:43Z

What changes were proposed in this pull request?

In previous work SPARK-21513, we has allowed MapType and ArrayType of MapTypes convert to a json string but only for Scala API. In this follow-up PR, we will make SparkSQL support it for PySpark and SparkR, too. We also fix some little bugs and comments of the previous work in this follow-up PR.

For PySpark

>>> data = [(1, {"name": "Alice"})]
>>> df = spark.createDataFrame(data, ("key", "value"))
>>> df.select(to_json(df.value).alias("json")).collect()
[Row(json=u'{"name":"Alice")']
>>> data = [(1, [{"name": "Alice"}, {"name": "Bob"}])]
>>> df = spark.createDataFrame(data, ("key", "value"))
>>> df.select(to_json(df.value).alias("json")).collect()
[Row(json=u'[{"name":"Alice"},{"name":"Bob"}]')]

For SparkR

# Converts a map into a JSON object
df2 <- sql("SELECT map('name', 'Bob')) as people")
df2 <- mutate(df2, people_json = to_json(df2$people))
# Converts an array of maps into a JSON array
df2 <- sql("SELECT array(map('name', 'Bob'), map('name', 'Alice')) as people")
df2 <- mutate(df2, people_json = to_json(df2$people))

How was this patch tested?

Add unit test cases.

cc @viirya @HyukjinKwon

HyukjinKwon · 2017-09-13T22:02:16Z

ok to test

SparkQA · 2017-09-13T22:07:31Z

Test build #81739 has finished for PR 19223 at commit 29e7323.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2017-09-13T23:33:49Z

R/pkg/R/functions.R

-#' \code{to_json}: Converts a column containing a \code{structType} or array of \code{structType}
-#' into a Column of JSON string. Resolving the Column can fail if an unsupported type is encountered.
+#' \code{to_json}: Converts a column containing a \code{structType}, array of \code{structType},
+#  a \code{mapType} or array of \code{mapType} into a Column of JSON string.


Looks ' is missed at the first #.

HyukjinKwon · 2017-09-13T23:49:16Z

python/pyspark/sql/functions.py

-    JSON string. Throws an exception, in the case of an unsupported type.
+    Converts a column containing a [[StructType]], [[ArrayType]] of [[StructType]]s,
+    a [[MapType]] or [[ArrayType]] of [[MapType]] into a JSON string.
+    Throws an exception, in the case of an unsupported type.


While we are here, let's fix [[StructType]] to :class:`StructType` (and the same instances too) to make Python API documentation pretty.

HyukjinKwon · 2017-09-14T00:49:07Z

R/pkg/R/functions.R

+#'
+#' # Converts an array of maps into a JSON array
+#' df2 <- sql("SELECT array(map('name', 'Bob'), map('name', 'Alice')) as people")
+#' df2 <- mutate(df2, people_json = to_json(df2$people))


And.. missing closing parentheses for \dontrun{.

... meaning }

ok Thanks for careful review :)

viirya · 2017-09-14T04:30:27Z

sql/core/src/test/resources/sql-tests/results/json-functions.sql.out

      > SELECT to_json(array(named_struct('a', 1, 'b', 2));
       [{"a":1,"b":2}]
-      > SELECT to_json(map('a',named_struct('b',1)));
+      > SELECT to_json(map('a', named_struct('b', 1)));


I think you committed unrelated change?

Or you forget to commit json-functions.sql?

umm. I modified ExpressionDescription of StructsToJson at @HyukjinKwon 's suggestions which didn't be merged in last PR. Here's the test for describe function extended to_json, so I needed to regenerate the golden file for it. So this change isn't from json-functions.sql.

…goldmedal/spark into SPARK-21513-fp-PySaprkAndSparkR

viirya · 2017-09-14T06:20:27Z

python/pyspark/sql/functions.py

    """
-    Converts a column containing a [[StructType]] or [[ArrayType]] of [[StructType]]s into a
-    JSON string. Throws an exception, in the case of an unsupported type.
+    Converts a column containing a :class:`StructType, :class:`ArrayType` of :class:`StructType`s,


Missing a ` after StructType.

viirya · 2017-09-14T06:21:49Z

LGTM except for one comment left.

SparkQA · 2017-09-14T07:04:45Z

Test build #81766 has finished for PR 19223 at commit af8d941.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-09-14T07:04:46Z

Test build #81761 has finished for PR 19223 at commit 158140e.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2017-09-14T07:07:38Z

retest this please.

SparkQA · 2017-09-14T09:59:39Z

Test build #81769 has finished for PR 19223 at commit af8d941.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

goldmedal · 2017-09-14T10:22:29Z

@HyukjinKwon @felixcheung @viirya
I has finished those change at your suggestions for this PR and it also passed all tests. Please take a look when you are available. Thanks :)

HyukjinKwon

LGTM @felixcheung, do you maybe have more comments?

HyukjinKwon · 2017-09-14T10:17:58Z

python/pyspark/sql/functions.py

    """
-    Converts a column containing a [[StructType]] or [[ArrayType]] of [[StructType]]s into a
-    JSON string. Throws an exception, in the case of an unsupported type.
+    Converts a column containing a :class:`StructType`, :class:`ArrayType` of :class:`StructType`s,


I believe :class:`StructType`s should be :class:`StructType`\\s. Currently, this is rendered as below:

ok, I'll modified it. Because I'm not really familiar with python, thanks for your suggestions. :)

That's fine :).

HyukjinKwon · 2017-09-14T10:18:29Z

python/pyspark/sql/functions.py

-    Converts a column containing a [[StructType]] or [[ArrayType]] of [[StructType]]s into a
-    JSON string. Throws an exception, in the case of an unsupported type.
+    Converts a column containing a :class:`StructType`, :class:`ArrayType` of :class:`StructType`s,
+    a :class:`MapType` or :class:`ArrayType` of :class:`MapType` into a JSON string.


`:class:`MapType` -> :class:`MapType`\\s for consistency.

SparkQA · 2017-09-14T10:44:19Z

Test build #81780 has finished for PR 19223 at commit 8a3a068.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-09-14T13:38:54Z

Test build #81781 has finished for PR 19223 at commit 66bc5b7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2017-09-14T14:28:13Z

LGTM

felixcheung · 2017-09-14T15:12:42Z

AppVeyor didn't run on this?

HyukjinKwon · 2017-09-15T00:30:06Z

D'oh, yes. I wonder why it was not triggered. I manually triggered via my account:

Build started: [SparkR] ALL
Diff: master...spark-test:8981A5F1-E2DC-4015-8266-12E9ADEE189B

goldmedal · 2017-09-15T00:39:12Z

@HyukjinKwon Thanks for triggering AppVeyor. In normal case, will AppVeyor be triggered automatically?

HyukjinKwon · 2017-09-15T00:42:25Z

Yes, when there are some changes in:

spark/appveyor.yml

Lines 29 to 35 in 828fab0

    
           - appveyor.yml 
        
           - dev/appveyor-install-dependencies.ps1 
        
           - R/ 
        
           - sql/core/src/main/scala/org/apache/spark/sql/api/r/ 
        
           - core/src/main/scala/org/apache/spark/api/r/ 
        
           - mllib/src/main/scala/org/apache/spark/ml/r/ 
        
           - core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala

It should run the R tests on Windows via AppVeyor.

goldmedal · 2017-09-15T00:44:33Z

ok. I got it. Thanks :)

HyukjinKwon · 2017-09-15T02:51:10Z

Looks passed fine. Let me merge this one.

viirya · 2017-09-15T02:52:57Z

Thanks @felixcheung @HyukjinKwon

HyukjinKwon · 2017-09-15T02:53:43Z

Merged to master.

goldmedal · 2017-09-15T02:56:42Z

Thanks @HyukjinKwon @felixcheung @viirya

goldmedal added 6 commits September 14, 2017 02:41

fix the coding style issue

071173c

fix the logic operator using

80fbb6b

add comments and test cases for sparkR

5e9a726

add comments and test cases for PySpark

6a3d374

fix some bug and comments

1f5b7cf

re-generate golden file

29e7323

HyukjinKwon reviewed Sep 13, 2017

View reviewed changes

HyukjinKwon reviewed Sep 14, 2017

View reviewed changes

viirya reviewed Sep 14, 2017

View reviewed changes

goldmedal added 3 commits September 14, 2017 13:31

fix some comments and test cases

124a90a

Merge branch 'SPARK-21513-fp-PySaprkAndSparkR' of https://github.com/…

e6dd4b9

…goldmedal/spark into SPARK-21513-fp-PySaprkAndSparkR

fix one comment style

158140e

viirya reviewed Sep 14, 2017

View reviewed changes

fix some comments

af8d941

HyukjinKwon reviewed Sep 14, 2017

View reviewed changes

fix comments

8a3a068

fix coding style issue

66bc5b7

asfgit closed this in a28728a Sep 15, 2017

goldmedal deleted the SPARK-21513-fp-PySaprkAndSparkR branch September 15, 2017 03:11

[SPARK-21513][SQL][FOLLOWUP] Allow UDF to_json support converting MapType to json for PySpark and SparkR #19223

[SPARK-21513][SQL][FOLLOWUP] Allow UDF to_json support converting MapType to json for PySpark and SparkR #19223

Uh oh!

Conversation

goldmedal commented Sep 13, 2017

What changes were proposed in this pull request?

For PySpark

For SparkR

How was this patch tested?

Uh oh!

HyukjinKwon commented Sep 13, 2017

Uh oh!

SparkQA commented Sep 13, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viirya commented Sep 14, 2017

Uh oh!

SparkQA commented Sep 14, 2017

Uh oh!

SparkQA commented Sep 14, 2017

Uh oh!

viirya commented Sep 14, 2017

Uh oh!

SparkQA commented Sep 14, 2017

Uh oh!

goldmedal commented Sep 14, 2017

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 14, 2017

Uh oh!

SparkQA commented Sep 14, 2017

Uh oh!

viirya commented Sep 14, 2017

Uh oh!

felixcheung commented Sep 14, 2017 via email

Uh oh!

HyukjinKwon commented Sep 15, 2017

Uh oh!

goldmedal commented Sep 15, 2017

Uh oh!

HyukjinKwon commented Sep 15, 2017

Uh oh!

goldmedal commented Sep 15, 2017

Uh oh!

HyukjinKwon commented Sep 15, 2017

Uh oh!

viirya commented Sep 15, 2017

Uh oh!

HyukjinKwon commented Sep 15, 2017