[SPARK-15084][PYTHON][SQL] Use builder pattern to create SparkSession in PySpark.

[dongjoon-hyun]

What changes were proposed in this pull request?

This is a python port of corresponding Scala builder pattern code. sql.py is modified as a target example case.

How was this patch tested?

Manual.

[dongjoon-hyun]

@rxin .
This is the initial commit to confirm the direction. Could you give me some advice?

[rxin]

we should update SparkSession's doc itself to indicate how to create it.

[dongjoon-hyun]

Yep. It's updated.

[rxin]

cc @davies can you take a look at the builder API?

[SparkQA]

Test build #57589 has finished for PR 12860 at commit 2b55814.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class Builder(object):

[SparkQA]

Test build #57592 has finished for PR 12860 at commit 07a926b.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #57593 has finished for PR 12860 at commit 1fe7fb0.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #57602 has finished for PR 12860 at commit 356bc85.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

We could create a builder here, then we can use it like this:

SparkSession.builder.master().getOrCreater()

[davies]

nvm, we also create a Builder every time in Scala.

[dongjoon-hyun]

Thank you for review, @davies . I'll update soon.

[dongjoon-hyun]

@davies . I addressed two comments, but I'm not sure about the first one.
We need to change ScalaSession.Builder first if we want to change.

[davies]

It's weird to use SQLContext to create an SparkSession, can't we create an SparkSession directly?

[davies]

We also use that in Scala, it's OK for now.

[andrewor14]

if we just call scala's getOrCreate here then we don't need to fix this in the future

[andrewor14]

Looks good otherwise

[SparkQA]

Test build #57649 has finished for PR 12860 at commit 6cd7740.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Thank you, @davies and @andrewor14 .
Ya, it's still evolving! No problem. After merging #12873 , I'll update accordingly again.

[rxin]

It's been merged!

[dongjoon-hyun]

Great! Thank you, @rxin

[SparkQA]

Test build #57683 has finished for PR 12860 at commit ac5bc68.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class Builder(object):

[SparkQA]

Test build #57684 has finished for PR 12860 at commit 589cba8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Hi, @davies and @andrewor14 . Now, it's updated.

• Add stop in SparkSession
• Update builder pattern according to the Scala versions.
  • One builder per one SparkSession.
• Clean up docs.

For the calling Scala's .getOrCreate, I prefer to keep Python builder similar to Scala one. After updating Scala builder, I hope to update this in the same manner easily.

[andrewor14]

Thanks, merging into master 2.0

[dongjoon-hyun]

Thank you, @andrewor14 !