Skip to content

[SPARK-12629] [SparkR] Fixes for DataFrame saveAsTable method #10580

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 9 commits into from
Closed

[SPARK-12629] [SparkR] Fixes for DataFrame saveAsTable method #10580

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
62872f1
Some fixes for saveAsTable method
NarineK Jan 4, 2016
d55cc13
Fix merge conflicts
NarineK Jan 4, 2016
8948f98
Merge branch 'master' of https://github.com/apache/spark into sparkrS…
NarineK Jan 5, 2016
86c1432
change the signature
NarineK Jan 6, 2016
7de1894
Merge branch 'master' of https://github.com/apache/spark into sparkrS…
NarineK Jan 6, 2016
faa4971
fixed style check errors
NarineK Jan 7, 2016
fdd38c5
Added the changes to write.df and saveDF too
NarineK Jan 20, 2016
8abf4c1
fixing merge conflict
NarineK Jan 21, 2016
3c24314
fixed stylecheck error
NarineK Jan 21, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 17 additions & 6 deletions R/pkg/R/DataFrame.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1997,7 +1997,13 @@ setMethod("write.df",
signature(df = "DataFrame", path = "character"),
function(df, path, source = NULL, mode = "error", ...){
if (is.null(source)) {
sqlContext <- get(".sparkRSQLsc", envir = .sparkREnv)
if (exists(".sparkRSQLsc", envir = .sparkREnv)) {
sqlContext <- get(".sparkRSQLsc", envir = .sparkREnv)
} else if (exists(".sparkRHivesc", envir = .sparkREnv)) {
sqlContext <- get(".sparkRHivesc", envir = .sparkREnv)
} else {
stop("sparkRHive or sparkRSQL context has to be specified")
}
source <- callJMethod(sqlContext, "getConf", "spark.sql.sources.default",
"org.apache.spark.sql.parquet")
}
Expand Down Expand Up @@ -2055,13 +2061,18 @@ setMethod("saveDF",
#' saveAsTable(df, "myfile")
#' }
Copy link
Contributor

@sun-rui sun-rui Jan 5, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the same problem exists in write.df()? Could you extract the following logic into a util function, and update write.df() also to use it?

Copy link
Member

@felixcheung felixcheung Jan 5, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #9192 :)

Copy link
Contributor Author

@NarineK NarineK Jan 5, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@felixcheung , @sun-rui , @shivaram : that's a good idea, I can use "getSqlContext()" method proposed in https://github.com/apache/spark/pull/9192/files#diff-b11442485f6b77bf47b58b4747321638R36

I guess I need to wait that pull request to go in and then I can merge it with mine .

Copy link
Contributor Author

@NarineK NarineK Jan 5, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@felixcheung, it seems that you have it for saveAsTable too. After merge I can add my test cases ...

setMethod("saveAsTable",
signature(df = "DataFrame", tableName = "character", source = "character",
mode = "character"),
signature(df = "DataFrame", tableName = "character"),
function(df, tableName, source = NULL, mode="error", ...){
if (is.null(source)) {
sqlContext <- get(".sparkRSQLsc", envir = .sparkREnv)
source <- callJMethod(sqlContext, "getConf", "spark.sql.sources.default",
"org.apache.spark.sql.parquet")
if (exists(".sparkRSQLsc", envir = .sparkREnv)) {
sqlContext <- get(".sparkRSQLsc", envir = .sparkREnv)
} else if (exists(".sparkRHivesc", envir = .sparkREnv)) {
sqlContext <- get(".sparkRHivesc", envir = .sparkREnv)
} else {
stop("sparkRHive or sparkRSQL context has to be specified")
}
source <- callJMethod(sqlContext, "getConf", "spark.sql.sources.default",
"org.apache.spark.sql.parquet")
}
jmode <- convertToJSaveMode(mode)
options <- varargsToEnv(...)
Expand Down
12 changes: 10 additions & 2 deletions R/pkg/R/generics.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -539,7 +539,7 @@ setGeneric("sampleBy", function(x, col, fractions, seed) { standardGeneric("samp

#' @rdname saveAsTable
#' @export
setGeneric("saveAsTable", function(df, tableName, source, mode, ...) {
setGeneric("saveAsTable", function(df, tableName, source = NULL, mode = "error", ...) {
Copy link
Contributor

@sun-rui sun-rui Jan 5, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

setGeneric("saveAsTable", function(df, tableName, source = NULL, mode = "error", ...) { is better? since it indicates more precise signature.

Copy link
Contributor Author

@NarineK NarineK Jan 5, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, that works! thanks for the suggestion!

standardGeneric("saveAsTable")
})

Expand All @@ -552,7 +552,15 @@ setGeneric("transform", function(`_data`, ...) {standardGeneric("transform") })

#' @rdname write.df
#' @export
setGeneric("saveDF", function(df, path, ...) { standardGeneric("saveDF") })
setGeneric("write.df", function(df, path, source = NULL, mode = "error", ...) {
standardGeneric("write.df")
})

#' @rdname write.df
#' @export
setGeneric("saveDF", function(df, path, source = NULL, mode = "error", ...) {
standardGeneric("saveDF")
})

#' @rdname write.json
#' @export
Expand Down
15 changes: 14 additions & 1 deletion R/pkg/inst/tests/testthat/test_sparkSQL.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -953,8 +953,21 @@ test_that("test HiveContext", {
df3 <- sql(hiveCtx, "select * from json2")
expect_is(df3, "DataFrame")
expect_equal(count(df3), 3)

unlink(jsonPath2)

hivetestDataPath <- tempfile(pattern="sparkr-test", fileext=".tmp")
invisible(saveAsTable(df, "hivetestbl", path = hivetestDataPath))
df4 <- sql(hiveCtx, "select * from hivetestbl")
expect_is(df4, "DataFrame")
expect_equal(count(df4), 3)
unlink(hivetestDataPath)

parquetDataPath <- tempfile(pattern="sparkr-test", fileext=".tmp")
invisible(saveAsTable(df, "parquetest", "parquet", mode="overwrite", path=parquetDataPath))
df5 <- sql(hiveCtx, "select * from parquetest")
expect_is(df5, "DataFrame")
expect_equal(count(df5), 3)
unlink(parquetDataPath)
})

test_that("column operators", {
Expand Down