Skip to content

[SPARK-12600][SQL] Remove deprecated methods in Spark SQL #10559

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 11 commits into from
Closed

[SPARK-12600][SQL] Remove deprecated methods in Spark SQL #10559

wants to merge 11 commits into from

Conversation

@rxin
Copy link
Contributor

@rxin rxin commented Jan 3, 2016

No description provided.

rxin
rxin reviewed Jan 3, 2016
View reviewed changes
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/LegacyTypeStringParser.scala
Copy link
Contributor Author

@rxin rxin Jan 3, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is moved from DataType.scala. It is kept here so Spark 2.0 can still read parquet files written by Spark 1.1.

@SparkQA
Copy link

SparkQA commented Jan 3, 2016

Test build #48598 has finished for PR 10559 at commit 4d7105b.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin rxin force-pushed the remove-deprecated-sql branch from 4d7105b to 5a83d42 Compare January 4, 2016 00:59
@SparkQA
Copy link

SparkQA commented Jan 4, 2016

Test build #48622 has finished for PR 10559 at commit c4e37ae.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jan 4, 2016

Test build #48623 has finished for PR 10559 at commit 4df345e.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin rxin force-pushed the remove-deprecated-sql branch from 4df345e to aeaaeb4 Compare January 4, 2016 05:30
rxin
rxin reviewed Jan 4, 2016
View reviewed changes
python/pyspark/sql/readwriter.py
Copy link
Contributor Author

@rxin rxin Jan 4, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @davies this fixed a bug before with read.json.

@SparkQA
Copy link

SparkQA commented Jan 4, 2016

Test build #48638 has finished for PR 10559 at commit aeaaeb4.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jan 4, 2016

Test build #48639 has finished for PR 10559 at commit ecdd18c.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jan 4, 2016

Test build #48642 has finished for PR 10559 at commit f3f2821.

  • This patch fails SparkR unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@marmbrus
Copy link
Contributor

marmbrus commented Jan 4, 2016

LGTM

rxin
rxin reviewed Jan 4, 2016
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
Copy link
Contributor Author

@rxin rxin Jan 4, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this one is actually useful for both R / Python, and required for data sources that don't support paths. So I'm just removing the todo to remove it.

davies
davies reviewed Jan 4, 2016
View reviewed changes
sql/catalyst/src/main/scala/org/apache/spark/sql/types/DecimalType.scala
Copy link
Contributor

@davies davies Jan 4, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should keep this checking somewhere

@davies
Copy link
Contributor

davies commented Jan 4, 2016

LGTM except one comment.

@rxin rxin changed the title [SPARK-12600][SQL] Remove deprecated methods in Spark SQL - WIP [SPARK-12600][SQL] Remove deprecated methods in Spark SQL Jan 4, 2016
@SparkQA
Copy link

SparkQA commented Jan 5, 2016

Test build #48684 has finished for PR 10559 at commit 5d5092d.

  • This patch fails from timeout after a configured wait of 250m.
  • This patch merges cleanly.
  • This patch adds no public classes.

@felixcheung felixcheung mentioned this pull request Jan 5, 2016
Closed
1 task
@SparkQA
Copy link

SparkQA commented Jan 5, 2016

Test build #2315 has finished for PR 10559 at commit 5d5092d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@marmbrus
Copy link
Contributor

marmbrus commented Jan 5, 2016

To avoid conflicts I'm going to go ahead and merge this. @rxin can address comments in a followup.

jongyoul pushed a commit to jongyoul/spark that referenced this pull request Jan 5, 2016
@rxin @marmbrus
[SPARK-12600][SQL] Remove deprecated methods in Spark SQL
77ab49b 
Author: Reynold Xin <[email protected]>

Closes apache#10559 from rxin/remove-deprecated-sql.
@rxin rxin closed this Jan 5, 2016
@rxin rxin mentioned this pull request Jan 5, 2016
Closed
@SparkQA
Copy link

SparkQA commented Jan 5, 2016

Test build #2318 has finished for PR 10559 at commit 5d5092d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

asfgit pushed a commit that referenced this pull request Jan 5, 2016
@rxin
[SPARK-12600][SQL] follow up: add range check for DecimalType
b634901 
This addresses davies' code review feedback in #10559

Author: Reynold Xin <[email protected]>

Closes #10586 from rxin/remove-deprecated-sql-followup.
asfgit pushed a commit that referenced this pull request Jan 5, 2016
@felixcheung @rxin
[SPARK-12625][SPARKR][SQL] replace R usage of Spark SQL deprecated API
cc4d522 
rxin davies shivaram
Took save mode from my PR #10480, and move everything to writer methods. This is related to PR #10559

- [x] it seems jsonRDD() is broken, need to investigate - this is not a public API though; will look into some more tonight. (fixed)

Author: felixcheung <[email protected]>

Closes #10584 from felixcheung/rremovedeprecated.
@zsxwing zsxwing mentioned this pull request Jan 29, 2016
Closed
asfgit pushed a commit that referenced this pull request Jan 29, 2016
@zsxwing
[SPARK-13082][PYSPARK] Backport the fix of 'read.json(rdd)' in #10559 …
84dab72 
…to branch-1.6

SPARK-13082 actually fixed by  #10559. However, it's a big PR and not backported to 1.6. This PR just backported the fix of 'read.json(rdd)' to branch-1.6.

Author: Shixiong Zhu <[email protected]>

Closes #10988 from zsxwing/json-rdd.
@HyukjinKwon HyukjinKwon mentioned this pull request Mar 30, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@rxin @SparkQA @marmbrus @davies