[SPARK-12222] [Core] Deserialize RoaringBitmap using Kryo serializer throw Buffer underflow exception #10213
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
/cc @davies |
|
Test build #47396 has finished for PR 10213 at commit
|
Contributor
|
LGTM, merging into master and 1.6, thanks! |
asfgit
pushed a commit
that referenced
this pull request
Dec 9, 2015
…throw Buffer underflow exception Jira: https://issues.apache.org/jira/browse/SPARK-12222 Deserialize RoaringBitmap using Kryo serializer throw Buffer underflow exception: ``` com.esotericsoftware.kryo.KryoException: Buffer underflow. at com.esotericsoftware.kryo.io.Input.require(Input.java:156) at com.esotericsoftware.kryo.io.Input.skip(Input.java:131) at com.esotericsoftware.kryo.io.Input.skip(Input.java:264) ``` This is caused by a bug of kryo's `Input.skip(long count)`(EsotericSoftware/kryo#119) and we call this method in `KryoInputDataInputBridge`. Instead of upgrade kryo's version, this pr bypass the kryo's `Input.skip(long count)` by directly call another `skip` method in kryo's Input.java(https://github.com/EsotericSoftware/kryo/blob/kryo-2.21/src/com/esotericsoftware/kryo/io/Input.java#L124), i.e. write the bug-fixed version of `Input.skip(long count)` in KryoInputDataInputBridge's `skipBytes` method. more detail link to #9748 (comment) Author: Fei Wang <[email protected]> Closes #10213 from scwf/patch-1. (cherry picked from commit 3934562) Signed-off-by: Davies Liu <[email protected]>
drcrallen
pushed a commit
to metamx/spark
that referenced
this pull request
Dec 16, 2015
…throw Buffer underflow exception Jira: https://issues.apache.org/jira/browse/SPARK-12222 Deserialize RoaringBitmap using Kryo serializer throw Buffer underflow exception: ``` com.esotericsoftware.kryo.KryoException: Buffer underflow. at com.esotericsoftware.kryo.io.Input.require(Input.java:156) at com.esotericsoftware.kryo.io.Input.skip(Input.java:131) at com.esotericsoftware.kryo.io.Input.skip(Input.java:264) ``` This is caused by a bug of kryo's `Input.skip(long count)`(EsotericSoftware/kryo#119) and we call this method in `KryoInputDataInputBridge`. Instead of upgrade kryo's version, this pr bypass the kryo's `Input.skip(long count)` by directly call another `skip` method in kryo's Input.java(https://github.com/EsotericSoftware/kryo/blob/kryo-2.21/src/com/esotericsoftware/kryo/io/Input.java#L124), i.e. write the bug-fixed version of `Input.skip(long count)` in KryoInputDataInputBridge's `skipBytes` method. more detail link to apache#9748 (comment) Author: Fei Wang <[email protected]> Closes apache#10213 from scwf/patch-1.
asfgit
pushed a commit
that referenced
this pull request
Dec 28, 2015
…hrow Buffer underflow exception Since we only need to implement `def skipBytes(n: Int)`, code in #10213 could be simplified. davies scwf Author: Daoyuan Wang <[email protected]> Closes #10253 from adrian-wang/kryo. (cherry picked from commit a6d3853) Signed-off-by: Kousuke Saruta <[email protected]>
asfgit
pushed a commit
that referenced
this pull request
Dec 28, 2015
…hrow Buffer underflow exception Since we only need to implement `def skipBytes(n: Int)`, code in #10213 could be simplified. davies scwf Author: Daoyuan Wang <[email protected]> Closes #10253 from adrian-wang/kryo.
|
does this problem have been solved in 1.6.0? (p.mobile_id, 'model', 'MOBILE_MODEL') as mobile_model from analytics_device_profile_zh10 p limit 1; mobileAttributeCahceMap (enterprise.hive.udfs.ConvertDeviceModel) |
Contributor
|
@highmoutain This patch was included in 1.6.0. Your problem looks like not related to this one. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Jira: https://issues.apache.org/jira/browse/SPARK-12222
Deserialize RoaringBitmap using Kryo serializer throw Buffer underflow exception:
This is caused by a bug of kryo's
Input.skip(long count)(EsotericSoftware/kryo#119) and we call this method inKryoInputDataInputBridge.Instead of upgrade kryo's version, this pr bypass the kryo's
Input.skip(long count)by directly call anotherskipmethod in kryo's Input.java(https://github.com/EsotericSoftware/kryo/blob/kryo-2.21/src/com/esotericsoftware/kryo/io/Input.java#L124), i.e. write the bug-fixed version ofInput.skip(long count)in KryoInputDataInputBridge'sskipBytesmethod.more detail link to #9748 (comment)