[SPARK-5577] Python udf for DataFrame #4351

davies · 2015-02-04T02:06:17Z

No description provided.

SparkQA · 2015-02-04T02:12:31Z

Test build #26705 has started for PR 4351 at commit c6d0d59.

This patch merges cleanly.

AmplabJenkins · 2015-02-04T02:22:23Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26703/
Test FAILed.

SparkQA · 2015-02-04T03:54:42Z

Test build #26705 has finished for PR 4351 at commit c6d0d59.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class FPGrowthModel[Item: ClassTag](
- class Dsl(object):
- class ExamplePointUDT(UserDefinedType):
- class SQLTests(ReusedPySparkTestCase):

AmplabJenkins · 2015-02-04T03:54:46Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26705/
Test PASSed.

davies · 2015-02-04T05:39:36Z

python/pyspark/sql.py

there are two sample().

oops mistakenly deleted my own comment

SparkQA · 2015-02-04T06:42:52Z

Test build #26727 has started for PR 4351 at commit 7bccc3b.

This patch merges cleanly.

SparkQA · 2015-02-04T08:21:05Z

Test build #26727 has finished for PR 4351 at commit 7bccc3b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-04T08:21:09Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26727/
Test PASSed.

SparkQA · 2015-02-04T08:32:53Z

Test build #26737 has started for PR 4351 at commit 462b334.

This patch merges cleanly.

rxin · 2015-02-04T08:33:05Z

python/pyspark/sql.py

we should use simplestring (which isn't available yet...) we can change it in the future

SparkQA · 2015-02-04T09:02:36Z

Test build #26740 has started for PR 4351 at commit f99b2e1.

This patch merges cleanly.

SparkQA · 2015-02-04T10:09:51Z

Test build #26737 has finished for PR 4351 at commit 462b334.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-04T10:09:55Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26737/
Test PASSed.

SparkQA · 2015-02-04T10:28:35Z

Test build #26740 has finished for PR 4351 at commit f99b2e1.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-04T10:28:39Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26740/
Test FAILed.

SparkQA · 2015-02-04T17:57:56Z

Test build #26765 has started for PR 4351 at commit f0a3121.

This patch merges cleanly.

SparkQA · 2015-02-04T18:10:14Z

Test build #582 has started for PR 4351 at commit f0a3121.

This patch merges cleanly.

SparkQA · 2015-02-04T18:12:30Z

Test build #26766 has started for PR 4351 at commit 440f769.

This patch merges cleanly.

SparkQA · 2015-02-04T19:39:47Z

Test build #26765 has finished for PR 4351 at commit f0a3121.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class UserDefinedFunction(object):

AmplabJenkins · 2015-02-04T19:39:51Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26765/
Test PASSed.

SparkQA · 2015-02-04T19:51:13Z

Test build #582 has finished for PR 4351 at commit f0a3121.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class UserDefinedFunction(object):

SparkQA · 2015-02-04T19:53:51Z

Test build #26766 has finished for PR 4351 at commit 440f769.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class UserDefinedFunction(object):

AmplabJenkins · 2015-02-04T19:53:56Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26766/
Test PASSed.

Conflicts: sql/core/src/main/scala/org/apache/spark/sql/Dsl.scala

SparkQA · 2015-02-04T20:22:52Z

Test build #26770 has started for PR 4351 at commit 34234d4.

This patch merges cleanly.

SparkQA · 2015-02-04T21:47:20Z

Test build #26770 has finished for PR 4351 at commit 34234d4.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DirectKafkaInputDStreamCheckpointData extends DStreamCheckpointData(this)
- class KafkaCluster(val kafkaParams: Map[String, String]) extends Serializable
- case class LeaderOffset(host: String, port: Int, offset: Long)
- class KafkaRDDPartition(
- trait HasOffsetRanges
- class UserDefinedFunction(object):

AmplabJenkins · 2015-02-04T21:47:25Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26770/
Test FAILed.

SparkQA · 2015-02-04T21:57:49Z

Test build #26778 has started for PR 4351 at commit d250692.

This patch merges cleanly.

SparkQA · 2015-02-04T23:38:35Z

Test build #26778 has finished for PR 4351 at commit d250692.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class UserDefinedFunction(object):

AmplabJenkins · 2015-02-04T23:38:38Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26778/
Test PASSed.

davies · 2015-02-04T23:53:38Z

@rxin I think this is ready.

rxin · 2015-02-04T23:56:18Z

Thanks. Merged.

Author: Davies Liu <[email protected]> Closes #4351 from davies/python_udf and squashes the following commits: d250692 [Davies Liu] fix conflict 34234d4 [Davies Liu] Merge branch 'master' of github.com:apache/spark into python_udf 440f769 [Davies Liu] address comments f0a3121 [Davies Liu] track life cycle of broadcast f99b2e1 [Davies Liu] address comments 462b334 [Davies Liu] Merge branch 'master' of github.com:apache/spark into python_udf 7bccc3b [Davies Liu] python udf 58dee20 [Davies Liu] clean up (cherry picked from commit dc101b0) Signed-off-by: Reynold Xin <[email protected]>

davies force-pushed the python_udf branch from 1e47664 to c6d0d59 Compare February 4, 2015 02:07

davies reviewed Feb 4, 2015
View reviewed changes

Davies Liu added 2 commits February 3, 2015 21:54

clean up

58dee20

python udf

7bccc3b

davies force-pushed the python_udf branch from c6d0d59 to 7bccc3b Compare February 4, 2015 06:38

davies changed the title ~~[WIP] [SPARK-5577] Python udf for DataFrame~~ [SPARK-5577] Python udf for DataFrame Feb 4, 2015

Merge branch 'master' of github.com:apache/spark into python_udf

462b334

rxin reviewed Feb 4, 2015
View reviewed changes

python/pyspark/sql.py

Copy link

Contributor

rxin Feb 4, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should use simplestring (which isn't available yet...) we can change it in the future

address comments

f99b2e1

track life cycle of broadcast

f0a3121

davies force-pushed the python_udf branch from 5be6368 to f0a3121 Compare February 4, 2015 17:54

address comments

440f769

Merge branch 'master' of github.com:apache/spark into python_udf

34234d4

Conflicts: sql/core/src/main/scala/org/apache/spark/sql/Dsl.scala

fix conflict

d250692

asfgit closed this in dc101b0 Feb 4, 2015

[SPARK-5577] Python udf for DataFrame #4351

[SPARK-5577] Python udf for DataFrame #4351

Uh oh!

Conversation

davies commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

AmplabJenkins commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

AmplabJenkins commented Feb 4, 2015

Uh oh!

davies Feb 4, 2015

Choose a reason for hiding this comment

Uh oh!

rxin Feb 4, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

AmplabJenkins commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

rxin Feb 4, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

AmplabJenkins commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

AmplabJenkins commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

AmplabJenkins commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

AmplabJenkins commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

AmplabJenkins commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

SparkQA commented Feb 4, 2015

Uh oh!

AmplabJenkins commented Feb 4, 2015

Uh oh!

davies commented Feb 4, 2015

Uh oh!

rxin commented Feb 4, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!