[SPARK-15863][SQL][DOC] Initial SQL programming guide update for Spark 2.0 #13592

liancheng · 2016-06-10T06:03:35Z

What changes were proposed in this pull request?

Initial SQL programming guide update for Spark 2.0. Contents like 1.6 to 2.0 migration guide are still incomplete.

We may also want to add more examples for Scala/Java Dataset typed transformations.

How was this patch tested?

N/A

rxin · 2016-06-10T06:10:10Z

docs/sql-programming-guide.md

-displayTitle: Spark SQL, DataFrames and Datasets Guide
-title: Spark SQL and DataFrames
+displayTitle: Spark SQL and Datasets Guide
+title: Spark SQL and Datasets


I'd keep DataFrame in the title since Python is using it.

SparkQA · 2016-06-10T06:24:15Z

Test build #60277 has finished for PR 13592 at commit 92f3f11.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-10T19:10:39Z

Test build #60309 has finished for PR 13592 at commit 819e109.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

WeichenXu123 · 2016-06-11T02:33:26Z

docs/sql-programming-guide.md

 {% highlight r %}
 # sc is an existing SparkContext.
-sqlContext <- sparkRSQL.init(sc)
+spark <- sparkRSQL.init(sc)


Currently, sparkRSQL.init call org.apache.spark.sql.api.r.SQLUtils.createSQLContext which return SQLContext object not SparkSession object. So here it seems we need to update the R api ?

R API is still in experimental status, and we haven't introduced SparkSession to SparkR yet.

cloud-fan · 2016-06-13T21:35:20Z

docs/sql-programming-guide.md

 by Spark SQL provide Spark with more information about the structure of both the data and the computation being performed. Internally,
 Spark SQL uses this extra information to perform extra optimizations. There are several ways to
-interact with Spark SQL including SQL, the DataFrames API and the Datasets API. When computing a result
+interact with Spark SQL including SQL and the Datasets API. When computing a result


how about , DataFrame API(python/R) and Dataset API(scala/java)

"Dataset API" instead of "Datasets API"?

SparkQA · 2016-06-16T09:48:29Z

Test build #60637 has finished for PR 13592 at commit 200a68c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-16T22:24:29Z

Test build #60664 has finished for PR 13592 at commit 4b3c4d3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-17T09:25:37Z

Test build #60698 has finished for PR 13592 at commit f413cbb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

liancheng · 2016-06-17T17:05:43Z

Thanks to everyone for the review!

maropu · 2016-06-18T05:54:23Z

@liancheng Is it worth adding two parameters spark.sql.files.maxPartitionBytes and spark.sql.files.openCostInBytes in Other Configuration Options? They are kinds of internal parameters though, it seems they are useful for the users that would like to control #partitions. https://issues.apache.org/jira/browse/SPARK-15894

felixcheung · 2016-06-20T08:56:39Z

Could you please update docs/sparkr.md too which cross link to, eg. sql-programming-guide.html#starting-point-sqlcontext

felixcheung · 2016-06-20T19:00:53Z

docs/sql-programming-guide.md

-`SQLContext` class, or one of its decedents. To create a basic `SQLContext`, all you need is a SparkContext.
+Unlike Scala, Java, and Python API, we haven't finished migrating `SQLContext` to `SparkSession` for SparkR yet, so
+the entry point into all relational functionality in SparkR is still the
+`SQLContext` class in Spark 2.0. To create a basic `SQLContext`, all you need is a `SparkContext`.


SparkSession support has been merged. The entrypoint is now sparkR.session()

And use of SQLContext is deprecated. Please see PR #13751
Both use of SparkContext and SQLContext in SparkR has been deprecated.

ah, just saw it. Great!

yhuai · 2016-06-20T21:49:21Z

Thanks! Let's get it in first and then we can revise it.

…k 2.0 ## What changes were proposed in this pull request? Initial SQL programming guide update for Spark 2.0. Contents like 1.6 to 2.0 migration guide are still incomplete. We may also want to add more examples for Scala/Java Dataset typed transformations. ## How was this patch tested? N/A Author: Cheng Lian <[email protected]> Closes #13592 from liancheng/sql-programming-guide-2.0. (cherry picked from commit 6df8e38) Signed-off-by: Yin Huai <[email protected]>

yhuai · 2016-06-20T21:54:11Z

@felixcheung I merged this one since I think it is better to make changes in parallel using this version as the foundation. Can you help on revising the R related doc? Thanks!

yhuai · 2016-06-20T21:59:03Z

docs/sql-programming-guide.md

 # Getting Started

-## Starting Point: SQLContext
+## Starting Point: SparkSession


The doc looks like this. I am not sure if there is a better way to improve this section (making it clear that SparkSession is not available in SparkR). @felixcheung @shivaram maybe you have better ideas?

nvm. Just saw https://github.com/apache/spark/pull/13592/files#r67746855

liancheng · 2016-06-20T23:49:58Z

@felixcheung Thanks for the review and your work on PR #13751! Was traveling during the weekend. Let's address these comments in follow-up PRs.

liancheng · 2016-06-21T02:55:09Z

@maropu Sorry for the late reply. Yea, adding description to these two options makes sense. Would you like to open a PR for this? Thanks!

maropu · 2016-06-21T02:55:55Z

@liancheng okay, I'll do that.

…ude sparkSession in R ## What changes were proposed in this pull request? Update doc as per discussion in PR #13592 ## How was this patch tested? manual shivaram liancheng Author: Felix Cheung <[email protected]> Closes #13799 from felixcheung/rsqlprogrammingguide. (cherry picked from commit 58f6e27) Signed-off-by: Cheng Lian <[email protected]>

…ude sparkSession in R ## What changes were proposed in this pull request? Update doc as per discussion in PR #13592 ## How was this patch tested? manual shivaram liancheng Author: Felix Cheung <[email protected]> Closes #13799 from felixcheung/rsqlprogrammingguide.

Initial SQL programming guide update for Spark 2.0

92f3f11

rxin reviewed Jun 10, 2016
View reviewed changes

Addresses PR comments

819e109

liancheng force-pushed the sql-programming-guide-2.0 branch from 10b3618 to 819e109 Compare June 10, 2016 18:50

liancheng mentioned this pull request Jun 11, 2016

[SPARK-15805][SQL][Documents] update sql programming guide for spark 2.0 including all scala/java/python code in the guide #13544

Closed

WeichenXu123 reviewed Jun 11, 2016
View reviewed changes

liancheng mentioned this pull request Jun 13, 2016

[SPARK-15925][SQL][SPARKR] Replaces registerTempTable with createOrReplaceTempView #13644

Closed

cloud-fan reviewed Jun 13, 2016
View reviewed changes

Fixes one more R sample code

4b3c4d3

More corrections

f413cbb

shivaram mentioned this pull request Jun 20, 2016

[SPARK-15159][SPARKR] SparkSession roxygen2 doc, programming guide, example updates #13751

Closed

felixcheung reviewed Jun 20, 2016
View reviewed changes

asfgit closed this in 6df8e38 Jun 20, 2016

yhuai reviewed Jun 20, 2016
View reviewed changes

liancheng deleted the sql-programming-guide-2.0 branch June 20, 2016 23:50

felixcheung mentioned this pull request Jun 21, 2016

[SPARK-15863][SQL][DOC][SPARKR] sql programming guide updates to include sparkSession in R #13799

Closed

[SPARK-15863][SQL][DOC] Initial SQL programming guide update for Spark 2.0 #13592

[SPARK-15863][SQL][DOC] Initial SQL programming guide update for Spark 2.0 #13592

Uh oh!

Conversation

liancheng commented Jun 10, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

rxin Jun 10, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 10, 2016

Uh oh!

SparkQA commented Jun 10, 2016

Uh oh!

WeichenXu123 Jun 11, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liancheng Jun 11, 2016

Choose a reason for hiding this comment

Uh oh!

cloud-fan Jun 13, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clockfly Jun 14, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 16, 2016

Uh oh!

SparkQA commented Jun 16, 2016

Uh oh!

SparkQA commented Jun 17, 2016

Uh oh!

liancheng commented Jun 17, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maropu commented Jun 18, 2016

Uh oh!

felixcheung commented Jun 20, 2016

Uh oh!

felixcheung Jun 20, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

felixcheung Jun 20, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yhuai Jun 20, 2016

Choose a reason for hiding this comment

Uh oh!

yhuai commented Jun 20, 2016

Uh oh!

yhuai commented Jun 20, 2016

Uh oh!

yhuai Jun 20, 2016

Choose a reason for hiding this comment

Uh oh!

yhuai Jun 20, 2016

Choose a reason for hiding this comment

Uh oh!

liancheng commented Jun 20, 2016

Uh oh!

liancheng commented Jun 21, 2016

Uh oh!

maropu commented Jun 21, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

liancheng commented Jun 10, 2016 •

edited

Loading

WeichenXu123 Jun 11, 2016 •

edited

Loading

cloud-fan Jun 13, 2016 •

edited

Loading

liancheng commented Jun 17, 2016 •

edited

Loading

felixcheung Jun 20, 2016 •

edited

Loading

felixcheung Jun 20, 2016 •

edited

Loading