Skip to content

[SPARK-20585][SPARKR] R generic hint support #17851

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from
Closed

[SPARK-20585][SPARKR] R generic hint support #17851

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
e21d51e
Initial implementation
zero323 May 3, 2017
261e5a6
Add since note
zero323 May 3, 2017
ee52b53
Fix style
zero323 May 4, 2017
a1e9233
Put hint generic in the right place
zero323 May 4, 2017
633b038
Remove empty line
zero323 May 4, 2017
3ed4d76
Adjust hint description
zero323 May 4, 2017
1183441
Adjust ... description
zero323 May 4, 2017
e6c6d82
Adujst description
zero323 May 4, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions R/pkg/NAMESPACE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,7 @@ exportMethods("arrange",
"group_by",
"groupBy",
"head",
"hint",
"insertInto",
"intersect",
"isLocal",
Expand Down
30 changes: 30 additions & 0 deletions R/pkg/R/DataFrame.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3715,3 +3715,33 @@ setMethod("rollup",
sgd <- callJMethod(x@sdf, "rollup", jcol)
groupedData(sgd)
})

#' hint
#'
#' Specifies execution plan hint and return a new SparkDataFrame.
#'
#' @param x a SparkDataFrame.
#' @param name a name of the hint.
#' @param ... optional parameters for the hint.
#' @return A SparkDataFrame.
#' @family SparkDataFrame functions
#' @aliases hint,SparkDataFrame,character-method
#' @rdname hint
#' @name hint
#' @export
#' @examples
#' \dontrun{
#' df <- createDataFrame(mtcars)
#' avg_mpg <- mean(groupBy(createDataFrame(mtcars), "cyl"), "mpg")
Copy link
Member

@felixcheung felixcheung May 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you recreated createDataFrame(mtcars) here, do you mean to use df from the line before?

Copy link
Member

@felixcheung felixcheung May 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or you want to show these are two different dataset? maybe it's worthwhile to comment that

Copy link
Member Author

@zero323 zero323 May 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to use different datasets to avoid aliasing and trivially equal warning. It works but it is confusing (note: equi-join syntax same as in Scala or Python would be great, and it shouldn't be that hard to add). Once we merge alias this shouldn't been an issue. Since we don't run it, I can add aliases now.

Copy link
Member

@felixcheung felixcheung May 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alias is going to 2.3, this is going to 2.2 - I think we leave this for now and can improve this in master later

Copy link
Member Author

@zero323 zero323 May 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also with alias it will be quite dense:

#' @examples
#' \dontrun{
#' # Set aliases to avoid ambiguity
#' df <- alias(createDataFrame(mtcars), "cars")
#' avg_mpg <- alias(mean(groupBy(createDataFrame(mtcars), "cyl"), "mpg"), "avg_mpg")
#'
#' head(join(
#'   df, hint(avg_mpg, "broadcast"), 
#'   column("cars.cyl") == column("avg_mpg.cyl")
#' ))
#' }

Copy link
Member

@felixcheung felixcheung May 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right - I think the example makes sense now but it might not be very obvious - for example,

createDataFrame(mtcars)
createDataFrame(mtcars)

vs

df <- createDataFrame(mtcars)
df

is not very subtle unless you know what Spark is doing differently here. This is why I suggested pointing out the need to have distinct "copies" of data

#'
#' head(join(df, hint(avg_mpg, "broadcast"), df$cyl == avg_mpg$cyl))
#' }
#' @note hint since 2.2.0
setMethod("hint",
signature(x = "SparkDataFrame", name = "character"),
function(x, name, ...) {
parameters <- list(...)
stopifnot(all(sapply(parameters, is.character)))
jdf <- callJMethod(x@sdf, "hint", name, parameters)
dataFrame(jdf)
})
4 changes: 4 additions & 0 deletions R/pkg/R/generics.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -576,6 +576,10 @@ setGeneric("group_by", function(x, ...) { standardGeneric("group_by") })
#' @export
setGeneric("groupBy", function(x, ...) { standardGeneric("groupBy") })

#' @rdname hint
#' @export
setGeneric("hint", function(x, name, ...) { standardGeneric("hint") })

#' @rdname insertInto
#' @export
setGeneric("insertInto", function(x, tableName, ...) { standardGeneric("insertInto") })
Expand Down
12 changes: 12 additions & 0 deletions R/pkg/inst/tests/testthat/test_sparkSQL.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2147,6 +2147,18 @@ test_that("join(), crossJoin() and merge() on a DataFrame", {

unlink(jsonPath2)
unlink(jsonPath3)

# Join with broadcast hint
df1 <- sql("SELECT * FROM range(10e10)")
df2 <- sql("SELECT * FROM range(10e10)")

execution_plan <- capture.output(explain(join(df1, df2, df1$id == df2$id)))
expect_false(any(grepl("BroadcastHashJoin", execution_plan)))

execution_plan_hint <- capture.output(
explain(join(df1, hint(df2, "broadcast"), df1$id == df2$id))
)
expect_true(any(grepl("BroadcastHashJoin", execution_plan_hint)))
Copy link
Member

@felixcheung felixcheung May 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome!

})

test_that("toJSON() on DataFrame", {
Expand Down