Skip to content

[SPARK-1955][GraphX]: VertexRDD can incorrectly assume index sharing #4705

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[SPARK-1955][GraphX]: VertexRDD can incorrectly assume index sharing #4705

wants to merge 1 commit into from

Conversation

@brennonyork
Copy link

@brennonyork brennonyork commented Feb 20, 2015

Fixes the issue whereby when VertexRDD's are diffed, innerJoined, or leftJoined and have different partition sizes they fail under the zipPartitions method. This fix tests whether the partitions are equal or not and, if not, will repartition the other to match the partition size of the calling VertexRDD.

@AmplabJenkins
Copy link

AmplabJenkins commented Feb 20, 2015

Can one of the admins verify this patch?

@brennonyork
Copy link
Author

brennonyork commented Feb 20, 2015

/cc @ankurdave could you review, thanks!

@ankurdave
Copy link
Contributor

ankurdave commented Feb 21, 2015

ok to test

@SparkQA
Copy link

SparkQA commented Feb 21, 2015

Test build #27804 has started for PR 4705 at commit 0882590.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 21, 2015

Test build #27804 has finished for PR 4705 at commit 0882590.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Feb 21, 2015

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27804/
Test PASSed.

@srowen
Copy link
Member

srowen commented Feb 21, 2015

LGTM although @ankurdave should probably have the final say.

@brennonyork
Copy link
Author

brennonyork commented Feb 23, 2015

@ankurdave any thoughts? This is blocking progress on a few other GraphX JIRA's (SPARK-4600, SPARK-5790), so whenever you get a chance that'd be great, thanks!

@brennonyork
Copy link
Author

brennonyork commented Feb 24, 2015

bump

@maropu
Copy link
Member

maropu commented Feb 25, 2015

@brennonyork I'll add unit tests for your patch before/after the patch merged.

@brennonyork
Copy link
Author

brennonyork commented Feb 25, 2015

@maropu thanks!

asfgit pushed a commit that referenced this pull request Feb 25, 2015
@ankurdave
[SPARK-1955][GraphX]: VertexRDD can incorrectly assume index sharing
8073767 
Fixes the issue whereby when VertexRDD's are `diff`ed, `innerJoin`ed, or `leftJoin`ed and have different partition sizes they fail under the `zipPartitions` method. This fix tests whether the partitions are equal or not and, if not, will repartition the other to match the partition size of the calling VertexRDD.

Author: Brennon York <[email protected]>

Closes #4705 from brennonyork/SPARK-1955 and squashes the following commits:

0882590 [Brennon York] updated to properly handle differently-partitioned vertexRDDs

(cherry picked from commit 9f603fc)
Signed-off-by: Ankur Dave <[email protected]>
@asfgit asfgit closed this in 9f603fc Feb 25, 2015
asfgit pushed a commit that referenced this pull request Feb 25, 2015
@ankurdave
[SPARK-1955][GraphX]: VertexRDD can incorrectly assume index sharing
00112ba 
Fixes the issue whereby when VertexRDD's are `diff`ed, `innerJoin`ed, or `leftJoin`ed and have different partition sizes they fail under the `zipPartitions` method. This fix tests whether the partitions are equal or not and, if not, will repartition the other to match the partition size of the calling VertexRDD.

Author: Brennon York <[email protected]>

Closes #4705 from brennonyork/SPARK-1955 and squashes the following commits:

0882590 [Brennon York] updated to properly handle differently-partitioned vertexRDDs

(cherry picked from commit 9f603fc)
Signed-off-by: Ankur Dave <[email protected]>
@ankurdave
Copy link
Contributor

ankurdave commented Feb 25, 2015

Thanks! Merged into master, branch-1.3, and branch-1.2.

@ankurdave ankurdave mentioned this pull request Feb 25, 2015
Closed
@brennonyork brennonyork mentioned this pull request Mar 14, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@brennonyork @AmplabJenkins @ankurdave @SparkQA @srowen @maropu