[SPARK-1958] Calling .collect() on a SchemaRDD should call executeCollect() on the underlying query plan. #939
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…lect() on the underlying query plan.
|
Merged build triggered. |
|
Merged build started. |
Contributor
|
LGTM. |
Contributor
|
Actually - please define the return type explicitly for public methods. |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15341/ |
Contributor
|
Actually a lot of tests are failing ... |
|
Merged build triggered. |
|
Merged build started. |
Contributor
Author
|
@rxin Shame... underestimated this issue and didn't run full test locally :( I think the problem is that |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
Contributor
|
Thanks. I've merged this in master & branch-1.0. |
asfgit
pushed a commit
that referenced
this pull request
Jun 2, 2014
…lect() on the underlying query plan. In cases like `Limit` and `TakeOrdered`, `executeCollect()` makes optimizations that `execute().collect()` will not. Author: Cheng Lian <[email protected]> Closes #939 from liancheng/spark-1958 and squashes the following commits: bdc4a14 [Cheng Lian] Copy rows to present immutable data to users 8250976 [Cheng Lian] Added return type explicitly for public API 192a25c [Cheng Lian] [SPARK-1958] Calling .collect() on a SchemaRDD should call executeCollect() on the underlying query plan. (cherry picked from commit d000ca9) Signed-off-by: Reynold Xin <[email protected]>
pdeyhim
pushed a commit
to pdeyhim/spark-1
that referenced
this pull request
Jun 25, 2014
…lect() on the underlying query plan. In cases like `Limit` and `TakeOrdered`, `executeCollect()` makes optimizations that `execute().collect()` will not. Author: Cheng Lian <[email protected]> Closes apache#939 from liancheng/spark-1958 and squashes the following commits: bdc4a14 [Cheng Lian] Copy rows to present immutable data to users 8250976 [Cheng Lian] Added return type explicitly for public API 192a25c [Cheng Lian] [SPARK-1958] Calling .collect() on a SchemaRDD should call executeCollect() on the underlying query plan.
xiliu82
pushed a commit
to xiliu82/spark
that referenced
this pull request
Sep 4, 2014
…lect() on the underlying query plan. In cases like `Limit` and `TakeOrdered`, `executeCollect()` makes optimizations that `execute().collect()` will not. Author: Cheng Lian <[email protected]> Closes apache#939 from liancheng/spark-1958 and squashes the following commits: bdc4a14 [Cheng Lian] Copy rows to present immutable data to users 8250976 [Cheng Lian] Added return type explicitly for public API 192a25c [Cheng Lian] [SPARK-1958] Calling .collect() on a SchemaRDD should call executeCollect() on the underlying query plan.
flyrain
pushed a commit
to flyrain/spark
that referenced
this pull request
Sep 21, 2021
…on (apache#939) ### What changes were proposed in this pull request? This PR adds support for defining distribution and ordering during table creation. ### Why are the changes needed? This change is needed for feature parity with ADT in Spark 2. ### Does this PR introduce any user-facing change? This PR adds new optional table creation clauses but it should only affect Iceberg users. ### How was this patch tested? This PR comes with new tests. More tests are in Iceberg.
flyrain
pushed a commit
to flyrain/spark
that referenced
this pull request
Sep 21, 2021
…e creation (apache#939)" This reverts commit 18c6755.
flyrain
pushed a commit
to flyrain/spark
that referenced
this pull request
Sep 21, 2021
* Add Iceberg as a dep * rdar://70004937 Rewrite row-level operations for Iceberg (apache#922) * rdar://72811621 Control distribution and ordering during table creation (apache#939) This PR adds support for defining distribution and ordering during table creation. This change is needed for feature parity with ADT in Spark 2. This PR adds new optional table creation clauses but it should only affect Iceberg users. This PR comes with new tests. More tests are in Iceberg.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In cases like
LimitandTakeOrdered,executeCollect()makes optimizations thatexecute().collect()will not.