[SPARK-14932][SQL] Allow DataFrame.replace() to replace values with None #16225

jiayue-zhang · 2016-12-09T05:33:51Z

What changes were proposed in this pull request?

Allow DataFrame.replace() to replace with None/null values.

How was this patch tested?

Python doctest and unit test. Scala unit test.

HyukjinKwon · 2016-12-09T15:10:48Z

Would we need a test for Scala too? I checked this by myself and it seems working fine with Scala though. I could argue that this affects the language-specific functions of both Python and Scala as a not strong opinion.

JoshRosen · 2016-12-09T17:38:37Z

sql/core/src/main/scala/org/apache/spark/sql/DataFrameNaFunctions.scala

-      case _ => replacement.map { case (k, v) => (convertToDouble(k), convertToDouble(v)) }
-    }
+    val replacementMap: Map[_, _] =
+      if (replacement.head._2 == null)


I think you can just write case null => here and thereby avoid the if else

I tried that but failed. Scala doesn't allow pattern matching with null, scala.Nothing and scala.Null

Oh, wait case null => works. I tried case v null which doesn't. Let me modify this. Thanks!

SparkQA · 2016-12-10T16:55:38Z

Test build #3490 has finished for PR 16225 at commit 0b15c8f.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-05-23T17:37:39Z

ok to test

gatorsmile · 2017-05-23T19:06:47Z

@bravo-zhang Could you resolve the conflicts? I will review it then. Thanks!

SparkQA · 2017-05-23T20:32:54Z

Test build #77261 has finished for PR 16225 at commit 2c532c3.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

jiayue-zhang · 2017-05-24T00:53:10Z

Thanks for taking a look, @gatorsmile The conflicts have been resolved.
I appreciate if @zero323 can take a look as well since you made improvement on this function recently.

SparkQA · 2017-05-24T02:30:45Z

Test build #77273 has finished for PR 16225 at commit 43fb6bd.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-05-24T05:06:07Z

Test build #77282 has finished for PR 16225 at commit b5424d9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

holdenk

Thanks for working on this, sorry its taken so long to review. I did a first read through the Python side and I've got two minor questions. Hopefully @zero323, @davie , or @gatorsmile can also have some time once the current release is finished to take a look.

holdenk · 2017-06-15T09:49:38Z

python/pyspark/sql/dataframe.py

                "Got {0}".format(type(to_replace)))

-        if not isinstance(value, valid_types) and not isinstance(to_replace, dict):
+        if not isinstance(value, valid_types) and value is not None \


So slightly jet lagged style question: would this be clearer if we just add type(None) to L1398? I know PEP8 says we should only use is is not for checking if something is none rather than depending on the implicit conversion to boolean -- but since really checking the type here we aren't really in danger of that. (This is just a suggestion to make it easier to read - if others think its easier to read this way thats fine :)). @davies ?

holdenk · 2017-06-15T09:50:25Z

python/pyspark/sql/dataframe.py

            to_replace = [to_replace]

-        if isinstance(value, (float, int, long, basestring)):
+        if isinstance(value, (float, int, long, basestring)) or value is None:


same as above

holdenk · 2017-06-15T09:58:10Z

python/pyspark/sql/dataframe.py

-        if not any(all_of_type(rep_dict.keys()) and all_of_type(rep_dict.values())
+        if not any(all_of_type(rep_dict.keys())
+                   and (all_of_type(rep_dict.values())
+                        or list(rep_dict.values()).count(None) == len(rep_dict))


In the Scala code null is allowed in to be the replacement value for some of the elements but not in the Python code. Is this intentional? If so we should document it clearly and expand on the error message bellow (otherwise we should make it more flexible).

gatorsmile · 2017-06-27T06:42:40Z

We are closing it due to inactivity. please do reopen if you want to push it forward. Thanks!

jiayue-zhang · 2017-08-02T16:03:25Z

@holdenk Thanks for review. I'll combine type(None) in the isinstance. I also made Scala and Python to accept null more generally and in the same way.
PR is reopened at: #18820
@gatorsmile please also take a look.

## What changes were proposed in this pull request? This PR proposes to close stale PRs, mostly the same instances with apache#18017 I believe the author in apache#14807 removed his account. Closes apache#7075 Closes apache#8927 Closes apache#9202 Closes apache#9366 Closes apache#10861 Closes apache#11420 Closes apache#12356 Closes apache#13028 Closes apache#13506 Closes apache#14191 Closes apache#14198 Closes apache#14330 Closes apache#14807 Closes apache#15839 Closes apache#16225 Closes apache#16685 Closes apache#16692 Closes apache#16995 Closes apache#17181 Closes apache#17211 Closes apache#17235 Closes apache#17237 Closes apache#17248 Closes apache#17341 Closes apache#17708 Closes apache#17716 Closes apache#17721 Closes apache#17937 Added: Closes apache#14739 Closes apache#17139 Closes apache#17445 Closes apache#18042 Closes apache#18359 Added: Closes apache#16450 Closes apache#16525 Closes apache#17738 Added: Closes apache#16458 Closes apache#16508 Closes apache#17714 Added: Closes apache#17830 Closes apache#14742 ## How was this patch tested? N/A Author: hyukjinkwon <[email protected]> Closes apache#18417 from HyukjinKwon/close-stale-pr.

[SPARK-14932][SQL] Allow DataFrame.replace() to replace values with None

2653750

Scala test for df.replace with null

2eac8b9

JoshRosen reviewed Dec 9, 2016

View reviewed changes

jiayue-zhang added 2 commits December 9, 2016 09:44

Use pattern matching for null case

7949292

Fix indentation

0b15c8f

Fix Python style check

2c532c3

jiayue-zhang added 2 commits May 23, 2017 17:09

Merge branch 'master' into spark-14932

5ab39cc

Improve scala doc and pyspark test

43fb6bd

Fix python3 dict.values() syntax

b5424d9

holdenk reviewed Jun 15, 2017

View reviewed changes

HyukjinKwon mentioned this pull request Jun 25, 2017

[INFRA] Close stale PRs #18417

Closed

asfgit closed this in b32bd00 Jun 27, 2017

jiayue-zhang mentioned this pull request Aug 2, 2017

[SPARK-14932][SQL] Allow DataFrame.replace() to replace values with None #18820

Closed

[SPARK-14932][SQL] Allow DataFrame.replace() to replace values with None #16225

[SPARK-14932][SQL] Allow DataFrame.replace() to replace values with None #16225

Uh oh!

Conversation

jiayue-zhang commented Dec 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

HyukjinKwon commented Dec 9, 2016

Uh oh!

JoshRosen Dec 9, 2016

Choose a reason for hiding this comment

Uh oh!

jiayue-zhang Dec 9, 2016

Choose a reason for hiding this comment

Uh oh!

jiayue-zhang Dec 9, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 10, 2016

Uh oh!

gatorsmile commented May 23, 2017

Uh oh!

gatorsmile commented May 23, 2017

Uh oh!

SparkQA commented May 23, 2017

Uh oh!

jiayue-zhang commented May 24, 2017

Uh oh!

SparkQA commented May 24, 2017

Uh oh!

SparkQA commented May 24, 2017

Uh oh!

holdenk left a comment

Choose a reason for hiding this comment

Uh oh!

holdenk Jun 15, 2017

Choose a reason for hiding this comment

Uh oh!

holdenk Jun 15, 2017

Choose a reason for hiding this comment

Uh oh!

holdenk Jun 15, 2017

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented Jun 27, 2017

Uh oh!

jiayue-zhang commented Aug 2, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jiayue-zhang commented Dec 9, 2016 •

edited

Loading