[SPARK-10914] UnsafeRow serialization breaks when two machines have different Oops size. #9030

rxin · 2015-10-08T18:35:51Z

UnsafeRow contains 3 pieces of information when pointing to some data in memory (an object, a base offset, and length). When the row is serialized with Java/Kryo serialization, the object layout in memory can change if two machines have different pointer width (Oops in JVM).

To reproduce, launch Spark using

MASTER=local-cluster[2,1,1024] bin/spark-shell --conf "spark.executor.extraJavaOptions=-XX:-UseCompressedOops"

And then run the following

scala> sql("select 1 xx").collect()

…ifferent Oops size. The problem is that UnsafeRow contains 3 pieces of information when pointing to some data in memory (an object, a base offset, and length). When the row is serialized with Java/Kryo serialization, the object layout in memory can change if two machines have different pointer width (Oops in JVM). To reproduce, launch Spark using MASTER=local-cluster[2,1,1024] bin/spark-shell --conf "spark.executor.extraJavaOptions=-XX:-UseCompressedOops" And then run the following scala> sql("select 1 xx").collect() (cherry picked from commit 157b2a818d3993b1321cc41fb7b30407bd13490b) Signed-off-by: Reynold Xin <[email protected]>

rxin · 2015-10-08T18:36:01Z

cc @davies and @JoshRosen

rxin · 2015-10-08T18:37:38Z

sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoin.scala

going to revert this

davies · 2015-10-08T18:49:39Z

LGTM, we have done this for UTF8String already (not support Kryo).

@cloud-fan Should we also do this for UnsafeArrayData?

andrewor14 · 2015-10-08T18:53:48Z

sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java

can this just call writeExternal(out)?

cloud-fan · 2015-10-08T18:57:17Z

I think we should apply this to unsafe array too.

SparkQA · 2015-10-08T20:03:57Z

Test build #1860 has finished for PR 9030 at commit 9b79e6f.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-10-08T20:47:08Z

Test build #43414 has finished for PR 9030 at commit 9b79e6f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public final class UnsafeRow extends MutableRow implements Externalizable, KryoSerializable

SparkQA · 2015-10-08T21:39:59Z

Test build #1861 has finished for PR 9030 at commit 9b79e6f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public final class UnsafeRow extends MutableRow implements Externalizable, KryoSerializable

yhuai · 2015-10-08T22:14:04Z

test this please

SparkQA · 2015-10-09T00:06:02Z

Test build #43434 has finished for PR 9030 at commit 9b79e6f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public final class UnsafeRow extends MutableRow implements Externalizable, KryoSerializable

rxin · 2015-10-09T00:25:30Z

Merging this in master & branch-1.5.

…ifferent Oops size. UnsafeRow contains 3 pieces of information when pointing to some data in memory (an object, a base offset, and length). When the row is serialized with Java/Kryo serialization, the object layout in memory can change if two machines have different pointer width (Oops in JVM). To reproduce, launch Spark using MASTER=local-cluster[2,1,1024] bin/spark-shell --conf "spark.executor.extraJavaOptions=-XX:-UseCompressedOops" And then run the following scala> sql("select 1 xx").collect() Author: Reynold Xin <[email protected]> Closes #9030 from rxin/SPARK-10914. (cherry picked from commit 84ea287) Signed-off-by: Reynold Xin <[email protected]>

…ines have different Oops size ## What changes were proposed in this pull request? ApproxCountDistinctForIntervals holds the UnsafeArrayData data to initialize endpoints. When the UnsafeArrayData is serialized with Java serialization, the BYTE_ARRAY_OFFSET in memory can change if two machines have different pointer width (Oops in JVM). This PR fixes this issue by using the same way in #9030 ## How was this patch tested? Manual test has been done in our tpcds environment and regarding unit test case has been added as well Closes #24317 from pengbo/SPARK-27406. Authored-by: mingbo_pb <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

## What changes were proposed in this pull request? Finish the rest work of #24317, #9030 a. Implement Kryo serialization for UnsafeArrayData b. fix UnsafeMapData Java/Kryo Serialization issue when two machines have different Oops size c. Move the duplicate code "getBytes()" to Utils. ## How was this patch tested? According Units has been added & tested Closes #24357 from pengbo/SPARK-27416_new. Authored-by: pengbo <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

Finish the rest work of apache#24317, apache#9030 a. Implement Kryo serialization for UnsafeArrayData b. fix UnsafeMapData Java/Kryo Serialization issue when two machines have different Oops size c. Move the duplicate code "getBytes()" to Utils. According Units has been added & tested Closes apache#24357 from pengbo/SPARK-27416_new. Authored-by: pengbo <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…erialization … This is a Spark 2.4.x backport of #24357 by pengbo. Original description follows below: --- ## What changes were proposed in this pull request? Finish the rest work of #24317, #9030 a. Implement Kryo serialization for UnsafeArrayData b. fix UnsafeMapData Java/Kryo Serialization issue when two machines have different Oops size c. Move the duplicate code "getBytes()" to Utils. ## How was this patch tested? According Units has been added & tested Closes #25223 from JoshRosen/SPARK-27416-2.4. Authored-by: pengbo <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…erialization … This is a Spark 2.4.x backport of apache#24357 by pengbo. Original description follows below: --- ## What changes were proposed in this pull request? Finish the rest work of apache#24317, apache#9030 a. Implement Kryo serialization for UnsafeArrayData b. fix UnsafeMapData Java/Kryo Serialization issue when two machines have different Oops size c. Move the duplicate code "getBytes()" to Utils. ## How was this patch tested? According Units has been added & tested Closes apache#25223 from JoshRosen/SPARK-27416-2.4. Authored-by: pengbo <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…erialization This is a Spark 2.3.x backport of a 2.4.x backport of apache#24357 by pengbo. Original description follows below: --- Finish the rest work of apache#24317, apache#9030 a. Implement Kryo serialization for UnsafeArrayData b. fix UnsafeMapData Java/Kryo Serialization issue when two machines have different Oops size c. Move the duplicate code "getBytes()" to Utils. According Units has been added & tested Authored-by: pengbo <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> Signed-off-by: Jason Altekruse <[email protected]>

rxin reviewed Oct 8, 2015
View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoin.scala Outdated

Copy link

Contributor Author

rxin Oct 8, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

going to revert this

Revert BroadcastJoin change.

9b79e6f

andrewor14 reviewed Oct 8, 2015
View reviewed changes

sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java

Copy link

Contributor

andrewor14 Oct 8, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can this just call writeExternal(out)?

asfgit closed this in 84ea287 Oct 9, 2015

pengbo mentioned this pull request Apr 8, 2019

[Spark-27406][SQL]UnsafeArrayData serialization breaks when two machines have different Oops size #24317

Closed

JoshRosen mentioned this pull request Jul 22, 2019

[SPARK-27416][SQL][BRANCH-2.4] UnsafeMapData & UnsafeArrayData Kryo serialization … #25223

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-10914] UnsafeRow serialization breaks when two machines have different Oops size. #9030

[SPARK-10914] UnsafeRow serialization breaks when two machines have different Oops size. #9030

Uh oh!

rxin commented Oct 8, 2015

Uh oh!

rxin commented Oct 8, 2015

Uh oh!

rxin Oct 8, 2015

Uh oh!

davies commented Oct 8, 2015

Uh oh!

andrewor14 Oct 8, 2015

Uh oh!

cloud-fan commented Oct 8, 2015

Uh oh!

SparkQA commented Oct 8, 2015

Uh oh!

SparkQA commented Oct 8, 2015

Uh oh!

SparkQA commented Oct 8, 2015

Uh oh!

yhuai commented Oct 8, 2015

Uh oh!

SparkQA commented Oct 9, 2015

Uh oh!

rxin commented Oct 9, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

[SPARK-10914] UnsafeRow serialization breaks when two machines have different Oops size. #9030

[SPARK-10914] UnsafeRow serialization breaks when two machines have different Oops size. #9030

Uh oh!

Conversation

rxin commented Oct 8, 2015

Uh oh!

rxin commented Oct 8, 2015

Uh oh!

rxin Oct 8, 2015

Choose a reason for hiding this comment

Uh oh!

davies commented Oct 8, 2015

Uh oh!

andrewor14 Oct 8, 2015

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Oct 8, 2015

Uh oh!

SparkQA commented Oct 8, 2015

Uh oh!

SparkQA commented Oct 8, 2015

Uh oh!

SparkQA commented Oct 8, 2015

Uh oh!

yhuai commented Oct 8, 2015

Uh oh!

SparkQA commented Oct 9, 2015

Uh oh!

rxin commented Oct 9, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants