[SPARK-32437][CORE] Improve MapStatus deserialization speed with RoaringBitmap 0.9.0 #29233
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dongjoon-hyun
commented
Jul 25, 2020
| Deserialization 530 535 9 0.4 2651.1 0.3X | ||
| ------------------------------------------------------------------------------------------------------------------------- | ||
| Serialization 175 183 12 1.1 874.1 1.0X | ||
| Deserialization 458 462 6 0.4 2288.6 0.4X |
dongjoon-hyun
commented
Jul 25, 2020
| Deserialization 495 588 79 0.4 2476.7 0.3X | ||
| -------------------------------------------------------------------------------------------------------------------------- | ||
| Serialization 160 171 8 1.2 801.1 1.0X | ||
| Deserialization 453 484 38 0.4 2263.4 0.4X |
dongjoon-hyun
commented
Jul 25, 2020
| Deserialization 946 977 33 0.2 4730.2 1.8X | ||
| --------------------------------------------------------------------------------------------------------------------------- | ||
| Serialization 1641 1819 252 0.1 8204.1 1.0X | ||
| Deserialization 844 882 37 0.2 4219.7 1.9X |
dongjoon-hyun
commented
Jul 25, 2020
| Deserialization 929 941 19 0.2 4645.5 1.5X | ||
| ---------------------------------------------------------------------------------------------------------------------------- | ||
| Serialization 1360 1412 73 0.1 6799.3 1.0X | ||
| Deserialization 850 859 13 0.2 4249.9 1.6X |
dongjoon-hyun
commented
Jul 25, 2020
| Deserialization 943 970 32 0.2 4715.8 1.8X | ||
| --------------------------------------------------------------------------------------------------------------------------- | ||
| Serialization 1740 1903 231 0.1 8700.0 1.0X | ||
| Deserialization 872 888 24 0.2 4360.9 2.0X |
dongjoon-hyun
commented
Jul 25, 2020
| Deserialization 940 970 37 0.2 4699.1 1.5X | ||
| ---------------------------------------------------------------------------------------------------------------------------- | ||
| Serialization 1461 1469 11 0.1 7306.1 1.0X | ||
| Deserialization 871 889 22 0.2 4353.9 1.7X |
Member
Author
|
cc FYI, @HyukjinKwon since |
|
Test build #126538 has finished for PR 29233 at commit
|
srowen
reviewed
Jul 25, 2020
Member
srowen
left a comment
srowen
left a comment
There was a problem hiding this comment.
Looks good if it's a simple update and perf win.
Member
Author
|
Thank you, @srowen ! Merged to master. |
Member
Author
|
Oops.. I missed the dependency update and Jenkins passed without a dependency test. :( |
Member
|
LGTM except ^. |
holdenk
pushed a commit
to holdenk/spark
that referenced
this pull request
Oct 27, 2020
…ingBitmap 0.9.0 ### What changes were proposed in this pull request? This PR aims to speed up `MapStatus` deserialization by 5~18% with the latest RoaringBitmap `0.9.0` and new APIs. Note that we focus on `deserialization` time because `serialization` occurs once while `deserialization` occurs many times. ### Why are the changes needed? The current version is too old. We had better upgrade it to get the performance improvement and bug fixes. Although `MapStatusesSerDeserBenchmark` is synthetic, the benchmark result is updated with this patch. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the Jenkins or GitHub Action. Closes apache#29233 from dongjoon-hyun/SPARK-ROAR. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit f642234) Signed-off-by: Dongjoon Hyun <[email protected]>
otterc
pushed a commit
to linkedin/spark
that referenced
this pull request
Mar 22, 2023
…ingBitmap 0.9.0 This PR aims to speed up `MapStatus` deserialization by 5~18% with the latest RoaringBitmap `0.9.0` and new APIs. Note that we focus on `deserialization` time because `serialization` occurs once while `deserialization` occurs many times. The current version is too old. We had better upgrade it to get the performance improvement and bug fixes. Although `MapStatusesSerDeserBenchmark` is synthetic, the benchmark result is updated with this patch. No. Pass the Jenkins or GitHub Action. Closes apache#29233 from dongjoon-hyun/SPARK-ROAR. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> Ref: LIHADOOP-56788 RB=2401504 BUG=LIHADOOP-56788 G=spark-reviewers R=vsowrira,minyang,chsingh,yezhou,mmuralid A=chsingh
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR aims to speed up
MapStatusdeserialization by 5~18% with the latest RoaringBitmap0.9.0and new APIs. Note that we focus ondeserializationtime becauseserializationoccurs once whiledeserializationoccurs many times.Why are the changes needed?
The current version is too old. We had better upgrade it to get the performance improvement and bug fixes.
Although
MapStatusesSerDeserBenchmarkis synthetic, the benchmark result is updated with this patch.Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass the Jenkins or GitHub Action.