[SPARK-5752][SQL] Don't implicitly convert RDDs directly to DataFrames #4556

rxin · 2015-02-12T07:55:46Z

The old implicit would convert RDDs directly to DataFrames, and that added too many methods.
toDataFrame -> toDF
Dsl -> functions
implicits moved into SQLContext.implicits
addColumn -> withColumn
renameColumn -> withColumnRenamed

Python changes:

toDataFrame -> toDF
Dsl -> functions package
addColumn -> withColumn
renameColumn -> withColumnRenamed
add toDF functions to RDD on SQLContext init
add flatMap to DataFrame

SparkQA · 2015-02-12T07:57:26Z

Test build #27332 has started for PR 4556 at commit ab58d66.

This patch merges cleanly.

SparkQA · 2015-02-12T09:36:51Z

Test build #27332 has finished for PR 4556 at commit ab58d66.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait DataFrame extends RDDApi[Row] with Serializable

AmplabJenkins · 2015-02-12T09:36:55Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27332/
Test PASSed.

SparkQA · 2015-02-12T19:42:41Z

Test build #27371 has started for PR 4556 at commit f755d21.

This patch merges cleanly.

SparkQA · 2015-02-12T21:22:57Z

Test build #27371 has finished for PR 4556 at commit f755d21.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-12T21:23:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27371/
Test PASSed.

AmplabJenkins · 2015-02-12T23:02:17Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27391/
Test FAILed.

rxin · 2015-02-12T23:23:23Z

Jenkins, retest this please.

AmplabJenkins · 2015-02-12T23:42:16Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27394/
Test FAILed.

SparkQA · 2015-02-13T00:02:56Z

Test build #27397 has started for PR 4556 at commit 81e7685.

This patch merges cleanly.

SparkQA · 2015-02-13T00:03:52Z

Test build #27397 has finished for PR 4556 at commit 81e7685.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-13T00:03:53Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27397/
Test FAILed.

SparkQA · 2015-02-13T00:07:33Z

Test build #27400 has started for PR 4556 at commit ef8c510.

This patch merges cleanly.

SparkQA · 2015-02-13T00:10:28Z

Test build #27400 has finished for PR 4556 at commit ef8c510.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-13T00:10:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27400/
Test FAILed.

SparkQA · 2015-02-13T00:12:28Z

Test build #27401 has started for PR 4556 at commit 9e7f89e.

This patch merges cleanly.

SparkQA · 2015-02-13T00:22:28Z

Test build #27402 has started for PR 4556 at commit 3858ad9.

This patch merges cleanly.

SparkQA · 2015-02-13T01:31:13Z

Test build #27401 has finished for PR 4556 at commit 9e7f89e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-13T01:31:16Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27401/
Test FAILed.

SparkQA · 2015-02-13T01:36:49Z

Test build #27402 has finished for PR 4556 at commit 3858ad9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-13T01:36:52Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27402/
Test FAILed.

SparkQA · 2015-02-13T01:57:27Z

Test build #27407 has started for PR 4556 at commit b401335.

This patch merges cleanly.

- Dsl -> functions - implicits moved into SQLContext.implicits - addColumn -> withColumn - renameColumn -> withColumnRenamed

rxin · 2015-02-13T22:42:38Z

Jenkins, retest this please.

SparkQA · 2015-02-13T22:47:36Z

Test build #27459 has started for PR 4556 at commit a4aaf58.

This patch merges cleanly.

SparkQA · 2015-02-14T00:06:30Z

Test build #27459 has finished for PR 4556 at commit a4aaf58.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-14T00:06:34Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27459/
Test FAILed.

SparkQA · 2015-02-14T00:13:00Z

Test build #27465 has started for PR 4556 at commit ff5832c.

This patch merges cleanly.

SparkQA · 2015-02-14T01:42:47Z

Test build #27465 has finished for PR 4556 at commit ff5832c.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-14T01:42:52Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27465/
Test FAILed.

Conflicts: python/pyspark/sql/dataframe.py python/pyspark/sql/functions.py

SparkQA · 2015-02-14T03:42:29Z

Test build #27471 has started for PR 4556 at commit 61d3fca.

This patch merges cleanly.

SparkQA · 2015-02-14T05:01:12Z

Test build #27471 has finished for PR 4556 at commit 61d3fca.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-14T05:01:16Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27471/
Test FAILed.

SparkQA · 2015-02-14T05:07:32Z

Test build #27476 has started for PR 4556 at commit 5ef9910.

This patch merges cleanly.

SparkQA · 2015-02-14T07:00:24Z

Test build #27476 has finished for PR 4556 at commit 5ef9910.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-14T07:00:28Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27476/
Test PASSed.

- The old implicit would convert RDDs directly to DataFrames, and that added too many methods. - toDataFrame -> toDF - Dsl -> functions - implicits moved into SQLContext.implicits - addColumn -> withColumn - renameColumn -> withColumnRenamed Python changes: - toDataFrame -> toDF - Dsl -> functions package - addColumn -> withColumn - renameColumn -> withColumnRenamed - add toDF functions to RDD on SQLContext init - add flatMap to DataFrame Author: Reynold Xin <[email protected]> Author: Davies Liu <[email protected]> Closes #4556 from rxin/SPARK-5752 and squashes the following commits: 5ef9910 [Reynold Xin] More fix 61d3fca [Reynold Xin] Merge branch 'df5' of github.com:davies/spark into SPARK-5752 ff5832c [Reynold Xin] Fix python 749c675 [Reynold Xin] count(*) fixes. 5806df0 [Reynold Xin] Fix build break again. d941f3d [Reynold Xin] Fixed explode compilation break. fe1267a [Davies Liu] flatMap c4afb8e [Reynold Xin] style d9de47f [Davies Liu] add comment b783994 [Davies Liu] add comment for toDF e2154e5 [Davies Liu] schema() -> schema 3a1004f [Davies Liu] Dsl -> functions, toDF() fb256af [Reynold Xin] - toDataFrame -> toDF - Dsl -> functions - implicits moved into SQLContext.implicits - addColumn -> withColumn - renameColumn -> withColumnRenamed 0dd74eb [Reynold Xin] [SPARK-5752][SQL] Don't implicitly convert RDDs directly to DataFrames 97dd47c [Davies Liu] fix mistake 6168f74 [Davies Liu] fix test 1fc0199 [Davies Liu] fix test a075cd5 [Davies Liu] clean up, toPandas 663d314 [Davies Liu] add test for agg('*') 9e214d5 [Reynold Xin] count(*) fixes. 1ed7136 [Reynold Xin] Fix build break again. 921b2e3 [Reynold Xin] Fixed explode compilation break. 14698d4 [Davies Liu] flatMap ba3e12d [Reynold Xin] style d08c92d [Davies Liu] add comment 5c8b524 [Davies Liu] add comment for toDF a4e5e66 [Davies Liu] schema() -> schema d377fc9 [Davies Liu] Dsl -> functions, toDF() 6b3086c [Reynold Xin] - toDataFrame -> toDF - Dsl -> functions - implicits moved into SQLContext.implicits - addColumn -> withColumn - renameColumn -> withColumnRenamed 807e8b1 [Reynold Xin] [SPARK-5752][SQL] Don't implicitly convert RDDs directly to DataFrames (cherry picked from commit e98dfe6) Signed-off-by: Reynold Xin <[email protected]>

rxin force-pushed the SPARK-5752 branch from ab58d66 to f755d21 Compare February 12, 2015 19:37

rxin force-pushed the SPARK-5752 branch from 3858ad9 to b401335 Compare February 13, 2015 01:56

rxin and others added 6 commits February 12, 2015 18:22

[SPARK-5752][SQL] Don't implicitly convert RDDs directly to DataFrames

807e8b1

- toDataFrame -> toDF

6b3086c

- Dsl -> functions - implicits moved into SQLContext.implicits - addColumn -> withColumn - renameColumn -> withColumnRenamed

Dsl -> functions, toDF()

d377fc9

schema() -> schema

a4e5e66

add comment for toDF

5c8b524

add comment

d08c92d

Davies Liu and others added 7 commits February 13, 2015 14:37

add comment for toDF

b783994

add comment

d9de47f

style

c4afb8e

flatMap

fe1267a

Fixed explode compilation break.

d941f3d

Fix build break again.

5806df0

count(*) fixes.

749c675

Fix python

ff5832c

rxin force-pushed the SPARK-5752 branch from a4aaf58 to ff5832c Compare February 14, 2015 00:11

Merge branch 'df5' of github.com:davies/spark into SPARK-5752

61d3fca

Conflicts: python/pyspark/sql/dataframe.py python/pyspark/sql/functions.py

More fix

5ef9910

asfgit closed this in e98dfe6 Feb 14, 2015

[SPARK-5752][SQL] Don't implicitly convert RDDs directly to DataFrames #4556

[SPARK-5752][SQL] Don't implicitly convert RDDs directly to DataFrames #4556

Uh oh!

Conversation

rxin commented Feb 12, 2015

Uh oh!

SparkQA commented Feb 12, 2015

Uh oh!

SparkQA commented Feb 12, 2015

Uh oh!

AmplabJenkins commented Feb 12, 2015

Uh oh!

SparkQA commented Feb 12, 2015

Uh oh!

SparkQA commented Feb 12, 2015

Uh oh!

AmplabJenkins commented Feb 12, 2015

Uh oh!

AmplabJenkins commented Feb 12, 2015

Uh oh!

rxin commented Feb 12, 2015

Uh oh!

AmplabJenkins commented Feb 12, 2015

Uh oh!

SparkQA commented Feb 13, 2015

Uh oh!

SparkQA commented Feb 13, 2015

Uh oh!

AmplabJenkins commented Feb 13, 2015

Uh oh!

SparkQA commented Feb 13, 2015

Uh oh!

SparkQA commented Feb 13, 2015

Uh oh!

AmplabJenkins commented Feb 13, 2015

Uh oh!

SparkQA commented Feb 13, 2015

Uh oh!

SparkQA commented Feb 13, 2015

Uh oh!

SparkQA commented Feb 13, 2015

Uh oh!

AmplabJenkins commented Feb 13, 2015

Uh oh!

SparkQA commented Feb 13, 2015

Uh oh!

AmplabJenkins commented Feb 13, 2015

Uh oh!

SparkQA commented Feb 13, 2015

Uh oh!

rxin commented Feb 13, 2015

Uh oh!

SparkQA commented Feb 13, 2015

Uh oh!

SparkQA commented Feb 14, 2015

Uh oh!

AmplabJenkins commented Feb 14, 2015

Uh oh!

SparkQA commented Feb 14, 2015

Uh oh!

SparkQA commented Feb 14, 2015

Uh oh!

AmplabJenkins commented Feb 14, 2015

Uh oh!

SparkQA commented Feb 14, 2015

Uh oh!

SparkQA commented Feb 14, 2015

Uh oh!

AmplabJenkins commented Feb 14, 2015

Uh oh!

SparkQA commented Feb 14, 2015

Uh oh!

SparkQA commented Feb 14, 2015

Uh oh!

AmplabJenkins commented Feb 14, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects