[SPARK-15996][R] Fix R examples by removing deprecated functions #13714
Conversation
dongjoon-hyun
changed the title
|
Test build #60655 has finished for PR 13714 at commit
|
|
Test build #60656 has finished for PR 13714 at commit
|
|
Hi, @shivaram , @felixcheung , @sun-rui . |
|
LGTM. Thanks for the fix @dongjoon-hyun |
|
Merging this to master and branch-2.0 |
|
Thank you, @shivaram ! |
## What changes were proposed in this pull request?
Currently, R examples(`dataframe.R` and `data-manipulation.R`) fail like the following. We had better update them before releasing 2.0 RC. This PR updates them to use up-to-date APIs.
```bash
$ bin/spark-submit examples/src/main/r/dataframe.R
...
Warning message:
'createDataFrame(sqlContext...)' is deprecated.
Use 'createDataFrame(data, schema = NULL, samplingRatio = 1.0)' instead.
See help("Deprecated")
...
Warning message:
'read.json(sqlContext...)' is deprecated.
Use 'read.json(path)' instead.
See help("Deprecated")
...
Error: could not find function "registerTempTable"
Execution halted
```
## How was this patch tested?
Manual.
```
curl -LO http://s3-us-west-2.amazonaws.com/sparkr-data/flights.csv
bin/spark-submit examples/src/main/r/dataframe.R
bin/spark-submit examples/src/main/r/data-manipulation.R flights.csv
```
Author: Dongjoon Hyun <[email protected]>
Closes #13714 from dongjoon-hyun/SPARK-15996.
(cherry picked from commit a865f6e)
Signed-off-by: Shivaram Venkataraman <[email protected]>
|
Thanks for catching these! |
| # First, register the flights SparkDataFrame as a table | ||
| registerTempTable(flightsDF, "flightsTable") | ||
| destDF <- sql(sqlContext, "SELECT dest, cancelled FROM flightsTable") | ||
| createOrReplaceTempView(flightsDF, "flightsTable") |
There was a problem hiding this comment.
Hmm - it looks like we have made a breaking change? IMO it's ok to leave the registerTempTable - at least we should deprecate before removing it completely? @shivaram - It is SPARK-15925
There was a problem hiding this comment.
Hmm I assumed this was a breaking change in 2.0 in Spark SQL for Scala and Python as well. But looking at #12945 it looks like the call is only deprecated and its not a breaking change - In that case we should add it back with a similar deprecation warning
cc @liancheng
There was a problem hiding this comment.
At first I thought it might be OK to just replace registerTempTable with createOrReplaceTempView since SparkR is still in experimental status. But after double thinking about this, a deprecation makes more sense here. Will add back the removed function and deprecate it. Thanks!
There was a problem hiding this comment.
Thank you all for nice conclusion! :)
5 participants
What changes were proposed in this pull request?
Currently, R examples(
dataframe.Randdata-manipulation.R) fail like the following. We had better update them before releasing 2.0 RC. This PR updates them to use up-to-date APIs.How was this patch tested?
Manual.