[SPARK-15171][SQL]Deprecate registerTempTable and add dataset.createTempView #12945

clockfly · 2016-05-06T02:26:34Z

What changes were proposed in this pull request?

Deprecates registerTempTable and add dataset.createTempView, dataset.createOrReplaceTempView.

How was this patch tested?

Unit tests.

SparkQA · 2016-05-06T02:28:58Z

Test build #57947 has finished for PR 12945 at commit e4675b0.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-06T02:34:06Z

Test build #57949 has finished for PR 12945 at commit 8612976.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-06T02:38:53Z

Test build #57950 has finished for PR 12945 at commit ac4e502.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-06T03:13:53Z

Test build #57953 has finished for PR 12945 at commit 0bcbe8c.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-06T04:52:11Z

Test build #57956 has finished for PR 12945 at commit 029ce73.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-06T07:45:45Z

Test build #57964 has finished for PR 12945 at commit e17bd6d.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

clockfly · 2016-05-06T12:16:46Z

There are two places I am not very sure:

Should we also deprecate the python and R api registerTempTable?
Which we allow replacing table or not for API createTempView(table)? Now it calls createTempView(table, replaceIfExists = false), should we change it to createTempView(table, replaceIfExists = true)?

SparkQA · 2016-05-06T13:38:38Z

Test build #57986 has finished for PR 12945 at commit 5bd77e9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

liancheng · 2016-05-09T10:12:55Z

For question 1, I think we should also deprecate corresponding API in PySpark and SparkR. For question 2, I don't get it...

liancheng · 2016-05-09T10:24:23Z

I think we should also deprecate Catalog.dropTempTable and SQLContext.dropTempTable by Catalog.dropTempView and SQLContext.dropTempView.

yhuai · 2016-05-09T18:09:43Z

If possible, can you have a pr to just add the new API and deprecate the old API?

rxin · 2016-05-10T03:21:34Z

sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala

SQLContext -> SparkSession

SparkQA · 2016-05-10T07:00:05Z

Test build #58217 has finished for PR 12945 at commit 59654e6.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-10T07:16:27Z

Test build #58219 has finished for PR 12945 at commit d372575.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-10T07:51:26Z

Test build #58213 has finished for PR 12945 at commit 3f834a4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-10T07:56:08Z

Test build #58214 has finished for PR 12945 at commit d53ca63.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-10T09:12:01Z

Test build #58223 has finished for PR 12945 at commit 2c7b5f5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-10T11:02:55Z

Test build #58226 has finished for PR 12945 at commit cc74aaa.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2016-05-12T06:17:06Z

python/pyspark/sql/session.py


        return DataFrame(jdf, self._wrapped)

+    def _createTempView(self, viewName, dataFrame, replaceIfExists):


Why do we add this method?

liancheng · 2016-05-12T06:41:19Z

The PR description is inconsistent with changes made in the PR now. Please update.

SparkQA · 2016-05-12T06:56:18Z

Test build #58449 has finished for PR 12945 at commit a9bdd5b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

liancheng · 2016-05-12T07:07:00Z

sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala

+    noException should be thrownBy {
+      // Overrrides the existing temporary view with same name
+      dataset.createOrReplaceTempView("tempView")
+    }


In general, we try to avoid using fancy ScalaTest assertion facilities like this one. In this case, a simple call to createOrReplaceTempView and a comment saying that no exception should be thrown is enough.

liancheng · 2016-05-12T07:15:58Z

sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala

   * @group basic
   * @since 1.6.0
   */
+  @deprecated("Use createOrReplaceTempView(tableName) instead.", "2.0.0")


Nit: tableName => viewName

liancheng · 2016-05-12T07:36:59Z

LGTM except for one minor comment for the deprecation warning message.

liancheng · 2016-05-12T07:47:14Z

OK, I'm merging this to master and branch-2.0 since the last commit only changes deprecation message.

…TempView ## What changes were proposed in this pull request? Deprecates registerTempTable and add dataset.createTempView, dataset.createOrReplaceTempView. ## How was this patch tested? Unit tests. Author: Sean Zhong <[email protected]> Closes #12945 from clockfly/spark-15171. (cherry picked from commit 33c6eb5) Signed-off-by: Cheng Lian <[email protected]>

SparkQA · 2016-05-12T08:51:11Z

Test build #58456 has finished for PR 12945 at commit 977e01d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-12T09:08:11Z

Test build #58459 has finished for PR 12945 at commit b024e99.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-12T09:10:09Z

Test build #58457 has finished for PR 12945 at commit dd52802.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

….registerTempTable ## What changes were proposed in this pull request? Update the unit test code, examples, and documents to remove calls to deprecated method `dataset.registerTempTable`. ## How was this patch tested? This PR only changes the unit test code, examples, and comments. It should be safe. This is a follow up of PR #12945 which was merged. Author: Sean Zhong <[email protected]> Closes #13098 from clockfly/spark-15171-remove-deprecation.

….registerTempTable ## What changes were proposed in this pull request? Update the unit test code, examples, and documents to remove calls to deprecated method `dataset.registerTempTable`. ## How was this patch tested? This PR only changes the unit test code, examples, and comments. It should be safe. This is a follow up of PR #12945 which was merged. Author: Sean Zhong <[email protected]> Closes #13098 from clockfly/spark-15171-remove-deprecation. (cherry picked from commit 25b315e) Signed-off-by: Cheng Lian <[email protected]>

…placeTempView ## What changes were proposed in this pull request? This PR replaces `registerTempTable` with `createOrReplaceTempView` as a follow-up task of #12945. ## How was this patch tested? Existing SparkR tests. Author: Cheng Lian <[email protected]> Closes #13644 from liancheng/spark-15925-temp-view-for-r.

…placeTempView ## What changes were proposed in this pull request? This PR replaces `registerTempTable` with `createOrReplaceTempView` as a follow-up task of #12945. ## How was this patch tested? Existing SparkR tests. Author: Cheng Lian <[email protected]> Closes #13644 from liancheng/spark-15925-temp-view-for-r. (cherry picked from commit ced8d66) Signed-off-by: Shivaram Venkataraman <[email protected]>

## What changes were proposed in this pull request? after #12945, we renamed the `registerTempTable` to `createTempView`, as we do create a view actually. This PR renames `SQLTestUtils.withTempTable` to reflect this change. ## How was this patch tested? N/A Author: Wenchen Fan <[email protected]> Closes #14318 from cloud-fan/minor4.

after #12945, we renamed the `registerTempTable` to `createTempView`, as we do create a view actually. This PR renames `SQLTestUtils.withTempTable` to reflect this change. N/A Author: Wenchen Fan <[email protected]> Closes #14318 from cloud-fan/minor4. (cherry picked from commit 86c2752) Signed-off-by: Reynold Xin <[email protected]>

clockfly changed the title ~~[SPARK-15171][SQL]Deprecate registerTempTable and add dataset.createTempView~~ [SPARK-15171][SQL][WIP]Deprecate registerTempTable and add dataset.createTempView May 6, 2016

clockfly force-pushed the spark-15171 branch from e4675b0 to 8612976 Compare May 6, 2016 02:30

clockfly force-pushed the spark-15171 branch from 8612976 to ac4e502 Compare May 6, 2016 02:34

clockfly force-pushed the spark-15171 branch from ac4e502 to 0bcbe8c Compare May 6, 2016 03:09

clockfly force-pushed the spark-15171 branch from 0bcbe8c to 029ce73 Compare May 6, 2016 03:45

clockfly force-pushed the spark-15171 branch from 029ce73 to e17bd6d Compare May 6, 2016 05:55

rxin reviewed May 10, 2016
View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala Outdated

Copy link

Contributor

rxin May 10, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SQLContext -> SparkSession

clockfly force-pushed the spark-15171 branch 2 times, most recently from d53ca63 to 59654e6 Compare May 10, 2016 06:56

clockfly force-pushed the spark-15171 branch from 59654e6 to d372575 Compare May 10, 2016 07:05

clockfly force-pushed the spark-15171 branch from d372575 to 2c7b5f5 Compare May 10, 2016 07:33

cloud-fan reviewed May 12, 2016
View reviewed changes

update code

977e01d

liancheng reviewed May 12, 2016
View reviewed changes

update test.

dd52802

liancheng reviewed May 12, 2016
View reviewed changes

update doc

b024e99

asfgit closed this in 33c6eb5 May 12, 2016

clockfly mentioned this pull request May 13, 2016

[SPARK-15171][SQL] Remove the references to deprecated method dataset.registerTempTable #13098

Closed

liancheng mentioned this pull request Jun 13, 2016

[SPARK-15925][SQL][SPARKR] Replaces registerTempTable with createOrReplaceTempView #13644

Closed

shivaram mentioned this pull request Jun 16, 2016

[SPARK-15996][R] Fix R examples by removing deprecated functions #13714

Closed

cloud-fan mentioned this pull request Jul 22, 2016

[SPARK-16690][TEST] rename SQLTestUtils.withTempTable to withTempView #14318

Closed


		return DataFrame(jdf, self._wrapped)

		def _createTempView(self, viewName, dataFrame, replaceIfExists):

[SPARK-15171][SQL]Deprecate registerTempTable and add dataset.createTempView #12945

[SPARK-15171][SQL]Deprecate registerTempTable and add dataset.createTempView #12945

Uh oh!

Conversation

clockfly commented May 6, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented May 6, 2016

Uh oh!

SparkQA commented May 6, 2016

Uh oh!

SparkQA commented May 6, 2016

Uh oh!

SparkQA commented May 6, 2016

Uh oh!

SparkQA commented May 6, 2016

Uh oh!

SparkQA commented May 6, 2016

Uh oh!

clockfly commented May 6, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented May 6, 2016

Uh oh!

liancheng commented May 9, 2016

Uh oh!

liancheng commented May 9, 2016

Uh oh!

yhuai commented May 9, 2016

Uh oh!

rxin May 10, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 10, 2016

Uh oh!

SparkQA commented May 10, 2016

Uh oh!

SparkQA commented May 10, 2016

Uh oh!

SparkQA commented May 10, 2016

Uh oh!

SparkQA commented May 10, 2016

Uh oh!

SparkQA commented May 10, 2016

Uh oh!

cloud-fan May 12, 2016

Choose a reason for hiding this comment

Uh oh!

liancheng commented May 12, 2016

Uh oh!

SparkQA commented May 12, 2016

Uh oh!

liancheng May 12, 2016

Choose a reason for hiding this comment

Uh oh!

liancheng May 12, 2016

Choose a reason for hiding this comment

Uh oh!

liancheng commented May 12, 2016

Uh oh!

liancheng commented May 12, 2016

Uh oh!

SparkQA commented May 12, 2016

Uh oh!

SparkQA commented May 12, 2016

Uh oh!

SparkQA commented May 12, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

clockfly commented May 6, 2016 •

edited

Loading

clockfly commented May 6, 2016 •

edited

Loading