[SQL] Minor: Introduce SchemaRDD#aggregate() for simple aggregations #874
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rdd.aggregate(Sum('val)) is just shorthand for
rdd.groupBy()(Sum('val)), but seems be more natural than
doing a groupBy with no grouping expressions when you
really just want an aggregation over all rows.
Did not add a JavaSchemaRDD or Python API, as these seem to
be lacking in several other methods like groupBy() already --
leaving that cleanup for future patches.
aarondav
changed the title
Contributor
Author
There was a problem hiding this comment.
This example doesn't compile with the \ in there.
|
Merged build triggered. |
|
Merged build started. |
Contributor
There was a problem hiding this comment.
Can you say in the scaladoc that this is equivalent to groupBy()(...) ?
Contributor
|
LGTM other than the small addition to scaladoc. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
Contributor
Author
|
Added comment! |
|
Merged build triggered. |
|
Merged build started. |
Contributor
|
LGTM |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
Contributor
|
I've merged this into master & branch-1.0. |
asfgit
pushed a commit
that referenced
this pull request
May 26, 2014
```scala
rdd.aggregate(Sum('val))
```
is just shorthand for
```scala
rdd.groupBy()(Sum('val))
```
but seems be more natural than doing a groupBy with no grouping expressions when you really just want an aggregation over all rows.
Did not add a JavaSchemaRDD or Python API, as these seem to be lacking several other methods like groupBy() already -- leaving that cleanup for future patches.
Author: Aaron Davidson <[email protected]>
Closes #874 from aarondav/schemardd and squashes the following commits:
e9e68ee [Aaron Davidson] Add comment
db6afe2 [Aaron Davidson] Introduce SchemaRDD#aggregate() for simple aggregations
(cherry picked from commit c3576ff)
Signed-off-by: Reynold Xin <[email protected]>
pdeyhim
pushed a commit
to pdeyhim/spark-1
that referenced
this pull request
Jun 25, 2014
```scala
rdd.aggregate(Sum('val))
```
is just shorthand for
```scala
rdd.groupBy()(Sum('val))
```
but seems be more natural than doing a groupBy with no grouping expressions when you really just want an aggregation over all rows.
Did not add a JavaSchemaRDD or Python API, as these seem to be lacking several other methods like groupBy() already -- leaving that cleanup for future patches.
Author: Aaron Davidson <[email protected]>
Closes apache#874 from aarondav/schemardd and squashes the following commits:
e9e68ee [Aaron Davidson] Add comment
db6afe2 [Aaron Davidson] Introduce SchemaRDD#aggregate() for simple aggregations
xiliu82
pushed a commit
to xiliu82/spark
that referenced
this pull request
Sep 4, 2014
```scala
rdd.aggregate(Sum('val))
```
is just shorthand for
```scala
rdd.groupBy()(Sum('val))
```
but seems be more natural than doing a groupBy with no grouping expressions when you really just want an aggregation over all rows.
Did not add a JavaSchemaRDD or Python API, as these seem to be lacking several other methods like groupBy() already -- leaving that cleanup for future patches.
Author: Aaron Davidson <[email protected]>
Closes apache#874 from aarondav/schemardd and squashes the following commits:
e9e68ee [Aaron Davidson] Add comment
db6afe2 [Aaron Davidson] Introduce SchemaRDD#aggregate() for simple aggregations
wangyum
pushed a commit
that referenced
this pull request
May 26, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
is just shorthand for
but seems be more natural than doing a groupBy with no grouping expressions when you really just want an aggregation over all rows.
Did not add a JavaSchemaRDD or Python API, as these seem to be lacking several other methods like groupBy() already -- leaving that cleanup for future patches.