Skip to content

[SPARK-29499][CORE][PYSPARK] Add mapPartitionsWithIndex for RDDBarrier #26148

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 3 commits into from
Closed

[SPARK-29499][CORE][PYSPARK] Add mapPartitionsWithIndex for RDDBarrier #26148

wants to merge 3 commits into from

Conversation

@ConeyLiu
Copy link
Contributor

What changes were proposed in this pull request?

Add mapPartitionsWithIndex for RDDBarrier.

Why are the changes needed?

There is only one method in RDDBarrier. We often use the partition index as a label for the current partition. We need to get the index from TaskContext index in the method of mapPartitions which is not convenient.

Does this PR introduce any user-facing change?

No

How was this patch tested?

New UT.

@ConeyLiu
Copy link
Contributor Author

Hi @jiangxb1987, could you help to review this?

huaxingao
huaxingao reviewed Oct 18, 2019
View reviewed changes
huaxingao
huaxingao reviewed Oct 18, 2019
View reviewed changes
huaxingao
huaxingao reviewed Oct 18, 2019
View reviewed changes
huaxingao
huaxingao reviewed Oct 18, 2019
View reviewed changes
huaxingao
huaxingao reviewed Oct 18, 2019
View reviewed changes
@ConeyLiu
Copy link
Contributor Author

thanks @huaxingao for the review. The comments have addressed.

jiangxb1987
jiangxb1987 reviewed Oct 21, 2019
View reviewed changes
Copy link
Contributor

@jiangxb1987 jiangxb1987 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds good, also cc @mengxr

@jiangxb1987
Copy link
Contributor

LGTM

@huaxingao
Copy link
Contributor

LGTM
@jiangxb1987 Could you please start Jenkins?

@jiangxb1987
Copy link
Contributor

Retest this please.

@cloud-fan
Copy link
Contributor

ok to test

@SparkQA
Copy link

SparkQA commented Oct 23, 2019

Test build #112524 has finished for PR 26148 at commit 437aadf.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jiangxb1987
Copy link
Contributor

Thanks, merged to master!

@ConeyLiu
Copy link
Contributor Author

thanks all.

@ConeyLiu ConeyLiu deleted the barrier-index branch October 23, 2019 13:05
@zero323 zero323 mentioned this pull request Jan 7, 2020
Closed
47 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jiangxb1987 jiangxb1987 jiangxb1987 left review comments

@huaxingao huaxingao huaxingao left review comments

Assignees

No one assigned

Labels

PYSPARK SPARK CORE

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@ConeyLiu @jiangxb1987 @huaxingao @cloud-fan @SparkQA @dongjoon-hyun