[SPARK-15073][SQL] Hide SparkSession constructor from the public #12873
Conversation
|
Test build #57651 has finished for PR 12873 at commit
|
|
retest this please |
|
LGTM pending Jenkins. |
|
Test build #57655 has finished for PR 12873 at commit
|
| extends Serializable with Logging { self => | ||
|
|
||
| def this(sc: SparkContext) { | ||
| private[sql] def this(sc: SparkContext) { |
There was a problem hiding this comment.
actually is it possible to just make this private?
There was a problem hiding this comment.
we use this in SQLContext
There was a problem hiding this comment.
maybe we should change that and have sqlcontext always getting it from sparksession tiself?
There was a problem hiding this comment.
yeah I think the problem is that in getOrCreate right now we go the other way, i.e. getting the session from the context
|
Merging in master/branch-2.0. |
## What changes were proposed in this pull request? Users should use the builder pattern instead. ## How was this patch tested? Jenks. Author: Andrew Or <[email protected]> Closes #12873 from andrewor14/spark-session-constructor. (cherry picked from commit 588cac4) Signed-off-by: Reynold Xin <[email protected]>
| def createSparkSession(): SparkSession = { | ||
| if (SparkSession.hiveClassesArePresent) { | ||
| sparkSession = SparkSession.withHiveSupport(sparkContext) | ||
| sparkSession = SparkSession.builder.enableHiveSupport().getOrCreate() |
There was a problem hiding this comment.
They are not identical. The previous one will create a new spark session anyway, but the new one may not create a new spark session if sparkContext already exists. Then we can't guarantee the returned spark session supports hive.
…rkContext from it ## What changes were proposed in this pull request? see #12873 (comment). The problem is, if we create `SparkContext` first and then call `SparkSession.builder.enableHiveSupport().getOrCreate()`, we will reuse the existing `SparkContext` and the hive flag won't be set. ## How was this patch tested? verified it locally. Author: Wenchen Fan <[email protected]> Closes #12890 from cloud-fan/repl.
…rkContext from it ## What changes were proposed in this pull request? see #12873 (comment). The problem is, if we create `SparkContext` first and then call `SparkSession.builder.enableHiveSupport().getOrCreate()`, we will reuse the existing `SparkContext` and the hive flag won't be set. ## How was this patch tested? verified it locally. Author: Wenchen Fan <[email protected]> Closes #12890 from cloud-fan/repl.
4 participants
What changes were proposed in this pull request?
Users should use the builder pattern instead.
How was this patch tested?
Jenks.