[Spark-27406][SQL]UnsafeArrayData serialization breaks when two machines have different Oops size #24317

pengbo · 2019-04-08T12:04:28Z

What changes were proposed in this pull request?

ApproxCountDistinctForIntervals holds the UnsafeArrayData data to initialize endpoints. When the UnsafeArrayData is serialized with Java serialization, the BYTE_ARRAY_OFFSET in memory can change if two machines have different pointer width (Oops in JVM).

This PR fixes this issue by using the same way in #9030

How was this patch tested?

Manual test has been done in our tpcds environment and regarding unit test case has been added as well

…ve different Oops size

…7406

pengbo · 2019-04-08T12:05:00Z

@rxin Can you please have a look?

sandeep-katta · 2019-04-08T12:15:31Z

Same thing is required to handle for UnsafeMapData

sandeep-katta · 2019-04-08T12:17:33Z

sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java

 */

-public final class UnsafeArrayData extends ArrayData {
+public final class UnsafeArrayData extends ArrayData implements Externalizable {


implement KryoSerializable also

sandeep-katta · 2019-04-08T12:19:42Z

cc @cloud-fan , @srowen @HyukjinKwon @rxin

cloud-fan · 2019-04-08T14:08:56Z

can we create a common trait/utils to do serialization for all these 3 unsafe data structures (row, array, map)?

pengbo · 2019-04-08T14:17:26Z

@cloud-fan sounds reasonable.

However, row/array/map structs are slightly different which needs different way to serialize. getBytes() may be put in the "UnsafeSerializationHelper/Utils".

Do you have a better idea? Your comments will be appreciated.

HeartSaVioR · 2019-04-09T01:15:02Z

Maybe reconsidering #24050 or picking up the approach #24050 to here and minimize the boundary of area (like restricting to getBytes())?

cloud-fan · 2019-04-09T02:19:21Z

ok to test

cloud-fan · 2019-04-09T02:19:58Z

let's merge it first and think about code refactor later.

SparkQA · 2019-04-09T06:27:07Z

Test build #104417 has finished for PR 24317 at commit faec817.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class FileScanBuilder(

cloud-fan · 2019-04-09T07:43:20Z

thanks, merging to master!

cloud-fan · 2019-04-09T07:43:37Z

can you send another PR to 2.4? it conflicts. Thanks!

pengbo · 2019-04-09T09:29:22Z

Okay, It will be done tonight.
I will create another PR for UnsafeMapData and refactoring later this week.

This PR is the branch-2.4 version for #24317 Closes #24324 from pengbo/SPARK-27406-branch-2.4. Authored-by: mingbo_pb <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

## What changes were proposed in this pull request? Finish the rest work of #24317, #9030 a. Implement Kryo serialization for UnsafeArrayData b. fix UnsafeMapData Java/Kryo Serialization issue when two machines have different Oops size c. Move the duplicate code "getBytes()" to Utils. ## How was this patch tested? According Units has been added & tested Closes #24357 from pengbo/SPARK-27416_new. Authored-by: pengbo <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

Finish the rest work of apache#24317, apache#9030 a. Implement Kryo serialization for UnsafeArrayData b. fix UnsafeMapData Java/Kryo Serialization issue when two machines have different Oops size c. Move the duplicate code "getBytes()" to Utils. According Units has been added & tested Closes apache#24357 from pengbo/SPARK-27416_new. Authored-by: pengbo <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…erialization … This is a Spark 2.4.x backport of #24357 by pengbo. Original description follows below: --- ## What changes were proposed in this pull request? Finish the rest work of #24317, #9030 a. Implement Kryo serialization for UnsafeArrayData b. fix UnsafeMapData Java/Kryo Serialization issue when two machines have different Oops size c. Move the duplicate code "getBytes()" to Utils. ## How was this patch tested? According Units has been added & tested Closes #25223 from JoshRosen/SPARK-27416-2.4. Authored-by: pengbo <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

This PR is the branch-2.4 version for apache#24317 Closes apache#24324 from pengbo/SPARK-27406-branch-2.4. Authored-by: mingbo_pb <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

This PR is the branch-2.4 version for apache/spark#24317 Closes #24324 from pengbo/SPARK-27406-branch-2.4. Authored-by: mingbo_pb <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit 3352803)

…erialization … This is a Spark 2.4.x backport of apache#24357 by pengbo. Original description follows below: --- ## What changes were proposed in this pull request? Finish the rest work of apache#24317, apache#9030 a. Implement Kryo serialization for UnsafeArrayData b. fix UnsafeMapData Java/Kryo Serialization issue when two machines have different Oops size c. Move the duplicate code "getBytes()" to Utils. ## How was this patch tested? According Units has been added & tested Closes apache#25223 from JoshRosen/SPARK-27416-2.4. Authored-by: pengbo <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…erialization This is a Spark 2.3.x backport of a 2.4.x backport of apache#24357 by pengbo. Original description follows below: --- Finish the rest work of apache#24317, apache#9030 a. Implement Kryo serialization for UnsafeArrayData b. fix UnsafeMapData Java/Kryo Serialization issue when two machines have different Oops size c. Move the duplicate code "getBytes()" to Utils. According Units has been added & tested Authored-by: pengbo <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> Signed-off-by: Jason Altekruse <[email protected]>

This PR is the branch-2.4 version for apache/spark#24317 Closes #24324 from pengbo/SPARK-27406-branch-2.4. Authored-by: mingbo_pb <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

mingbo_pb added 2 commits April 8, 2019 19:51

SPARK-27406-UnsafeArrayData serialization breaks when two machines ha…

beb1ff9

…ve different Oops size

Merge branch 'master' of https://github.com/apache/spark into SPARK-2…

faec817

…7406

pengbo changed the title ~~[Spark 27406][SQL]UnsafeArrayData serialization breaks when two machines have different Oops size~~ [Spark-27406][SQL]UnsafeArrayData serialization breaks when two machines have different Oops size Apr 8, 2019

sandeep-katta reviewed Apr 8, 2019

View reviewed changes

cloud-fan closed this in 3e4cfe9 Apr 9, 2019

pengbo mentioned this pull request Apr 9, 2019

[Spark-27406][SQL]UnsafeArrayData serialization breaks when two machi… #24324

Closed

JoshRosen mentioned this pull request Jul 22, 2019

[SPARK-27416][SQL][BRANCH-2.4] UnsafeMapData & UnsafeArrayData Kryo serialization … #25223

Closed

[Spark-27406][SQL]UnsafeArrayData serialization breaks when two machines have different Oops size #24317

[Spark-27406][SQL]UnsafeArrayData serialization breaks when two machines have different Oops size #24317

Uh oh!

Conversation

pengbo commented Apr 8, 2019

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

pengbo commented Apr 8, 2019

Uh oh!

sandeep-katta commented Apr 8, 2019

Uh oh!

sandeep-katta Apr 8, 2019

Choose a reason for hiding this comment

Uh oh!

sandeep-katta commented Apr 8, 2019

Uh oh!

cloud-fan commented Apr 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pengbo commented Apr 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HeartSaVioR commented Apr 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloud-fan commented Apr 9, 2019

Uh oh!

cloud-fan commented Apr 9, 2019

Uh oh!

SparkQA commented Apr 9, 2019

Uh oh!

cloud-fan commented Apr 9, 2019

Uh oh!

cloud-fan commented Apr 9, 2019

Uh oh!

pengbo commented Apr 9, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

cloud-fan commented Apr 8, 2019 •

edited

Loading

pengbo commented Apr 8, 2019 •

edited

Loading

HeartSaVioR commented Apr 9, 2019 •

edited

Loading