[SPARK-15072][SQL][REPL][EXAMPLES] Remove SparkSession.withHiveSupport #12851

techaddict · 2016-05-02T22:59:35Z

What changes were proposed in this pull request?

Removing the withHiveSupport method of SparkSession, instead use enableHiveSupport

How was this patch tested?

ran tests locally

rxin · 2016-05-02T23:21:30Z

sql/hive/src/test/resources/regression-test-SPARK-8489/Main.scala

i think this one needs to be built?

Not sure I got you ? do you mean something like withSparkContext(sc) instead of getOrCreate()

In order to use this file, I think it needs to be built into a jar?

yeah we need to rebuild the jar. There's already a TODO on L31. Right now the corresponding test in HiveSparkSubmitSuite is ignored.

Since we not longer create a assembly jar, not sure how to create this jar easily(steps could be found here #11630).

let's just fix it later

andrewor14 · 2016-05-03T00:30:03Z

LGTM

SparkQA · 2016-05-03T01:22:07Z

Test build #57569 has finished for PR 12851 at commit 7a07671.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-03T03:35:00Z

Test build #2961 has finished for PR 12851 at commit 7a07671.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-05-03T18:40:46Z

Seems like some legitimate failure/?

techaddict · 2016-05-03T18:53:29Z

Jenkins retest this please.

SparkQA · 2016-05-03T21:17:47Z

Test build #57656 has finished for PR 12851 at commit 7a07671.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

techaddict · 2016-05-04T16:42:27Z

@rxin yupp failed tests are sql hive tests, waiting for #12890

andrewor14 · 2016-05-05T00:41:41Z

repl/scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala

    val builder = SparkSession.builder.config(conf)
    if (SparkSession.hiveClassesArePresent) {
-      sparkSession = builder.enableHiveSupport().getOrCreate()
+      sparkSession = SparkSession.builder.enableHiveSupport().getOrCreate()


wait, this is wrong now. We want to use the builder we already have

andrewor14 · 2016-05-05T00:44:58Z

@techaddict a general suggestion is to try to get the SparkContext from the SparkSession rather than the other way round. Right now we call getOrCreate() and assume there's a SparkContext nearby, which may be true but is kind of brittle and outright wrong in some cases.

SparkQA · 2016-05-05T00:53:57Z

Test build #57812 has finished for PR 12851 at commit 17e0703.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-05T07:44:38Z

Test build #57849 has finished for PR 12851 at commit 7ad494d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

techaddict · 2016-05-05T07:53:48Z

@rxin @andrewor14 Comments Addressed, now all tests are passing 🎉

andrewor14 · 2016-05-05T17:19:44Z

sql/hive/src/test/resources/regression-test-SPARK-8489/Main.scala

+      .setMaster("local")
+      .setAppName("testing")
+
+    val sparkSession = SparkSession.builder.enableHiveSupport().getOrCreate()


this should use the builder's conf method. By the way for this file I would revert the changes here and fix it in #12924 instead, since @dilipbiswal is actually rebuilding the jar there.

andrewor14 · 2016-05-05T17:22:22Z

@techaddict this is getting really close. For the regression test for SPARK-8489 I would just fix it later since this patch doesn't rebuild that jar anyway.

andrewor14 · 2016-05-05T19:15:46Z

LGTM

SparkQA · 2016-05-05T19:17:35Z

Test build #57904 has finished for PR 12851 at commit 5e464ad.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2016-05-05T21:34:52Z

Merging into master 2.0

## What changes were proposed in this pull request? Removing the `withHiveSupport` method of `SparkSession`, instead use `enableHiveSupport` ## How was this patch tested? ran tests locally Author: Sandeep Singh <[email protected]> Closes #12851 from techaddict/SPARK-15072. (cherry picked from commit ed6f3f8) Signed-off-by: Andrew Or <[email protected]>

…port in PySpark ## What changes were proposed in this pull request? This is a followup of #12851 Remove `SparkSession.withHiveSupport` in PySpark and instead use `SparkSession.builder. enableHiveSupport` ## How was this patch tested? Existing tests. Author: Sandeep Singh <[email protected]> Closes #13063 from techaddict/SPARK-15072-followup.

rxin reviewed May 2, 2016
View reviewed changes

[SPARK-15072][SQL][REPL][EXAMPLES] Remove SparkSession.withHiveSupport

17e0703

techaddict force-pushed the SPARK-15072 branch from 7a07671 to 17e0703 Compare May 4, 2016 22:48

andrewor14 reviewed May 5, 2016
View reviewed changes

Address comments

7ad494d

andrewor14 reviewed May 5, 2016
View reviewed changes

address comments

5e464ad

asfgit closed this in ed6f3f8 May 5, 2016

techaddict deleted the SPARK-15072 branch May 6, 2016 12:00

techaddict mentioned this pull request May 11, 2016

[SPARK-15072][SQL][PySpark] FollowUp: Remove SparkSession.withHiveSupport in PySpark #13063

Closed

[SPARK-15072][SQL][REPL][EXAMPLES] Remove SparkSession.withHiveSupport #12851

[SPARK-15072][SQL][REPL][EXAMPLES] Remove SparkSession.withHiveSupport #12851

Uh oh!

Conversation

techaddict commented May 2, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

rxin May 2, 2016

Choose a reason for hiding this comment

Uh oh!

techaddict May 2, 2016

Choose a reason for hiding this comment

Uh oh!

rxin May 3, 2016

Choose a reason for hiding this comment

Uh oh!

andrewor14 May 3, 2016

Choose a reason for hiding this comment

Uh oh!

techaddict May 3, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewor14 May 3, 2016

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented May 3, 2016

Uh oh!

SparkQA commented May 3, 2016

Uh oh!

SparkQA commented May 3, 2016

Uh oh!

rxin commented May 3, 2016

Uh oh!

techaddict commented May 3, 2016

Uh oh!

SparkQA commented May 3, 2016

Uh oh!

techaddict commented May 4, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrewor14 May 5, 2016

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented May 5, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented May 5, 2016

Uh oh!

SparkQA commented May 5, 2016

Uh oh!

techaddict commented May 5, 2016

Uh oh!

andrewor14 May 5, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented May 5, 2016

Uh oh!

andrewor14 commented May 5, 2016

Uh oh!

SparkQA commented May 5, 2016

Uh oh!

andrewor14 commented May 5, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

techaddict May 3, 2016 •

edited

Loading

techaddict commented May 4, 2016 •

edited

Loading

andrewor14 commented May 5, 2016 •

edited

Loading

andrewor14 May 5, 2016 •

edited

Loading