[SPARK-6876] [PySpark] [SQL] add DataFrame na.replace in pyspark #6003

adrian-wang · 2015-05-08T07:24:14Z

No description provided.

rxin · 2015-05-08T07:25:42Z

core/src/main/scala/org/apache/spark/api/python/PythonUtils.scala

toScalaMap?

sry I wasn't clear. I meant let's rename this to toScalaMap.

AmplabJenkins · 2015-05-08T07:27:11Z

Merged build triggered.

AmplabJenkins · 2015-05-08T07:27:17Z

Merged build started.

SparkQA · 2015-05-08T07:29:14Z

Test build #32215 has started for PR 6003 at commit 08a07ad.

SparkQA · 2015-05-08T07:30:49Z

Test build #32215 has finished for PR 6003 at commit 08a07ad.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-08T07:30:50Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-08T07:30:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32215/
Test FAILed.

AmplabJenkins · 2015-05-08T07:57:11Z

Merged build triggered.

AmplabJenkins · 2015-05-08T07:57:17Z

Merged build started.

SparkQA · 2015-05-08T07:57:31Z

Test build #32218 has started for PR 6003 at commit 484af9e.

SparkQA · 2015-05-08T09:37:36Z

Test build #32218 has finished for PR 6003 at commit 484af9e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-08T09:37:41Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-08T09:37:42Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32218/
Test PASSed.

rxin · 2015-05-10T08:34:26Z

python/pyspark/sql/dataframe.py

rename this _to_scala_map

AmplabJenkins · 2015-05-11T11:02:10Z

Merged build triggered.

AmplabJenkins · 2015-05-11T11:02:16Z

Merged build started.

SparkQA · 2015-05-11T11:03:57Z

Test build #32381 has started for PR 6003 at commit 04209b9.

SparkQA · 2015-05-11T12:43:44Z

Test build #32381 has finished for PR 6003 at commit 04209b9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-11T12:43:49Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-11T12:43:49Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32381/
Test PASSed.

rxin · 2015-05-11T22:15:14Z

Thanks. I finally had time to review this in detail. A few things we need:

There are lot of branches for testing various input types. We should test those as well in tests.py. (Not in docstring tests).
We should support the case where to_replace is just a dict, like Pandas does.
Ideally, we should also support the case where to_replace is a tuple, and value is just a single value. The semantics is that for all values in to_replace, just replace it with value.

rxin · 2015-05-11T22:15:27Z

python/pyspark/sql/dataframe.py

the error handling and input type checking are pretty nice!

AmplabJenkins · 2015-05-12T06:07:10Z

Build triggered.

AmplabJenkins · 2015-05-12T06:07:18Z

Build started.

SparkQA · 2015-05-12T06:09:10Z

Test build #32467 has started for PR 6003 at commit 2bb3b23.

rxin · 2015-05-12T06:10:09Z

LGTM. Can you update it so it can merge cleanly with master?

SparkQA · 2015-05-12T06:10:49Z

Test build #32467 has finished for PR 6003 at commit 2bb3b23.

This patch fails Python style tests.
This patch does not merge cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-12T06:10:50Z

Build finished. Test FAILed.

AmplabJenkins · 2015-05-12T06:10:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32467/
Test FAILed.

AmplabJenkins · 2015-05-12T06:27:11Z

Merged build triggered.

AmplabJenkins · 2015-05-12T06:27:19Z

Merged build started.

SparkQA · 2015-05-12T06:27:56Z

Test build #32469 has started for PR 6003 at commit 4a148f7.

SparkQA · 2015-05-12T07:38:10Z

Test build #32469 has finished for PR 6003 at commit 4a148f7.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-12T07:38:15Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-12T07:38:15Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32469/
Test FAILed.

AmplabJenkins · 2015-05-12T08:52:11Z

Merged build triggered.

AmplabJenkins · 2015-05-12T08:52:20Z

Merged build started.

SparkQA · 2015-05-12T08:54:23Z

Test build #32488 has started for PR 6003 at commit 672efba.

SparkQA · 2015-05-12T10:35:58Z

Test build #32488 has finished for PR 6003 at commit 672efba.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-12T10:36:03Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-12T10:36:03Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32488/
Test PASSed.

rxin · 2015-05-12T17:24:03Z

Thanks. Merging in!

Author: Daoyuan Wang <[email protected]> Closes #6003 from adrian-wang/pynareplace and squashes the following commits: 672efba [Daoyuan Wang] remove py2.7 feature 4a148f7 [Daoyuan Wang] to_replace support dict, value support single value, and add full tests 9e232e7 [Daoyuan Wang] rename scala map af0268a [Daoyuan Wang] remove na 63ac579 [Daoyuan Wang] add na.replace in pyspark (cherry picked from commit d86ce84) Signed-off-by: Reynold Xin <[email protected]>

Author: Daoyuan Wang <[email protected]> Closes apache#6003 from adrian-wang/pynareplace and squashes the following commits: 672efba [Daoyuan Wang] remove py2.7 feature 4a148f7 [Daoyuan Wang] to_replace support dict, value support single value, and add full tests 9e232e7 [Daoyuan Wang] rename scala map af0268a [Daoyuan Wang] remove na 63ac579 [Daoyuan Wang] add na.replace in pyspark

rxin reviewed May 8, 2015
View reviewed changes

rxin reviewed May 10, 2015
View reviewed changes

python/pyspark/sql/dataframe.py Outdated

Copy link

Contributor

rxin May 10, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rename this _to_scala_map

rxin reviewed May 11, 2015
View reviewed changes

python/pyspark/sql/dataframe.py Outdated

Copy link

Contributor

rxin May 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the error handling and input type checking are pretty nice!

adrian-wang added 4 commits May 11, 2015 23:23

add na.replace in pyspark

63ac579

remove na

af0268a

rename scala map

9e232e7

to_replace support dict, value support single value, and add full tests

4a148f7

adrian-wang force-pushed the pynareplace branch from 2bb3b23 to 4a148f7 Compare May 12, 2015 06:24

remove py2.7 feature

672efba

asfgit closed this in d86ce84 May 12, 2015

[SPARK-6876] [PySpark] [SQL] add DataFrame na.replace in pyspark #6003

[SPARK-6876] [PySpark] [SQL] add DataFrame na.replace in pyspark #6003

Uh oh!

Conversation

adrian-wang commented May 8, 2015

Uh oh!

rxin May 8, 2015

Choose a reason for hiding this comment

Uh oh!

adrian-wang May 8, 2015

Choose a reason for hiding this comment

Uh oh!

rxin May 10, 2015

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented May 8, 2015

Uh oh!

AmplabJenkins commented May 8, 2015

Uh oh!

SparkQA commented May 8, 2015

Uh oh!

SparkQA commented May 8, 2015

Uh oh!

AmplabJenkins commented May 8, 2015

Uh oh!

AmplabJenkins commented May 8, 2015

Uh oh!

AmplabJenkins commented May 8, 2015

Uh oh!

AmplabJenkins commented May 8, 2015

Uh oh!

SparkQA commented May 8, 2015

Uh oh!

SparkQA commented May 8, 2015

Uh oh!

AmplabJenkins commented May 8, 2015

Uh oh!

AmplabJenkins commented May 8, 2015

Uh oh!

rxin May 10, 2015

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented May 11, 2015

Uh oh!

AmplabJenkins commented May 11, 2015

Uh oh!

SparkQA commented May 11, 2015

Uh oh!

SparkQA commented May 11, 2015

Uh oh!

AmplabJenkins commented May 11, 2015

Uh oh!

AmplabJenkins commented May 11, 2015

Uh oh!

rxin commented May 11, 2015

Uh oh!

rxin May 11, 2015

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented May 12, 2015

Uh oh!

AmplabJenkins commented May 12, 2015

Uh oh!

SparkQA commented May 12, 2015

Uh oh!

rxin commented May 12, 2015

Uh oh!

SparkQA commented May 12, 2015

Uh oh!

AmplabJenkins commented May 12, 2015

Uh oh!

AmplabJenkins commented May 12, 2015

Uh oh!

AmplabJenkins commented May 12, 2015

Uh oh!

AmplabJenkins commented May 12, 2015

Uh oh!

SparkQA commented May 12, 2015

Uh oh!

SparkQA commented May 12, 2015