Skip to content

[SPARK-29576][Core] Use Spark's CompressionCodec for Ser/Deser of MapOutputStatus #26235

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

[SPARK-29576][Core] Use Spark's CompressionCodec for Ser/Deser of MapOutputStatus #26235

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
fcf64b2
Use Spark's CompressionCodec and optimize deserializeMapStatuses
dbtsai Oct 23, 2019
da276aa
fix import
dbtsai Oct 23, 2019
d0b1f4c
add comment
dbtsai Oct 23, 2019
013e7b2
Add JDK11
dongjoon-hyun Oct 23, 2019
6212855
Add JDK8
dongjoon-hyun Oct 23, 2019
e5acdbf
Merge pull request #8 from dongjoon-hyun/PR-26235
dbtsai Oct 23, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 19 additions & 19 deletions core/benchmarks/MapStatusesSerDeserBenchmark-jdk11-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,19 @@ OpenJDK 64-Bit Server VM 11.0.4+11-post-Ubuntu-1ubuntu218.04.3 on Linux 4.15.0-1
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 10 blocks w/ broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 205 213 13 1.0 1023.6 1.0X
Deserialization 908 939 27 0.2 4540.2 0.2X
Serialization 170 178 9 1.2 849.7 1.0X
Deserialization 530 535 9 0.4 2651.1 0.3X

Compressed Serialized MapStatus sizes: 400 bytes
Compressed Serialized MapStatus sizes: 411 bytes
Compressed Serialized Broadcast MapStatus sizes: 2 MB


OpenJDK 64-Bit Server VM 11.0.4+11-post-Ubuntu-1ubuntu218.04.3 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 10 blocks w/o broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 195 204 24 1.0 976.9 1.0X
Deserialization 913 940 33 0.2 4566.7 0.2X
Serialization 157 165 7 1.3 785.4 1.0X
Deserialization 495 588 79 0.4 2476.7 0.3X

Compressed Serialized MapStatus sizes: 2 MB
Compressed Serialized Broadcast MapStatus sizes: 0 bytes
Expand All @@ -24,43 +24,43 @@ OpenJDK 64-Bit Server VM 11.0.4+11-post-Ubuntu-1ubuntu218.04.3 on Linux 4.15.0-1
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 100 blocks w/ broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 616 619 3 0.3 3079.1 1.0X
Deserialization 936 954 22 0.2 4680.5 0.7X
Serialization 344 351 4 0.6 1720.4 1.0X
Deserialization 527 579 99 0.4 2635.9 0.7X

Compressed Serialized MapStatus sizes: 418 bytes
Compressed Serialized Broadcast MapStatus sizes: 14 MB
Compressed Serialized MapStatus sizes: 427 bytes
Compressed Serialized Broadcast MapStatus sizes: 13 MB


OpenJDK 64-Bit Server VM 11.0.4+11-post-Ubuntu-1ubuntu218.04.3 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 100 blocks w/o broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 586 588 3 0.3 2928.8 1.0X
Deserialization 929 933 4 0.2 4647.0 0.6X
Serialization 317 321 4 0.6 1583.8 1.0X
Deserialization 530 540 15 0.4 2648.3 0.6X

Compressed Serialized MapStatus sizes: 14 MB
Compressed Serialized MapStatus sizes: 13 MB
Compressed Serialized Broadcast MapStatus sizes: 0 bytes


OpenJDK 64-Bit Server VM 11.0.4+11-post-Ubuntu-1ubuntu218.04.3 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 1000 blocks w/ broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 4740 4916 249 0.0 23698.5 1.0X
Deserialization 1578 1597 27 0.1 7890.6 3.0X
Serialization 1738 1849 156 0.1 8692.0 1.0X
Deserialization 946 977 33 0.2 4730.2 1.8X

Compressed Serialized MapStatus sizes: 546 bytes
Compressed Serialized Broadcast MapStatus sizes: 123 MB
Compressed Serialized MapStatus sizes: 556 bytes
Compressed Serialized Broadcast MapStatus sizes: 121 MB


OpenJDK 64-Bit Server VM 11.0.4+11-post-Ubuntu-1ubuntu218.04.3 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 1000 blocks w/o broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 4492 4573 115 0.0 22458.3 1.0X
Deserialization 1533 1547 20 0.1 7664.8 2.9X
Serialization 1379 1432 76 0.1 6892.6 1.0X
Deserialization 929 941 19 0.2 4645.5 1.5X

Compressed Serialized MapStatus sizes: 123 MB
Compressed Serialized MapStatus sizes: 121 MB
Compressed Serialized Broadcast MapStatus sizes: 0 bytes


38 changes: 19 additions & 19 deletions core/benchmarks/MapStatusesSerDeserBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,19 @@ OpenJDK 64-Bit Server VM 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10 on Linux 4.15.
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 10 blocks w/ broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 236 245 18 0.8 1179.1 1.0X
Deserialization 842 885 37 0.2 4211.4 0.3X
Serialization 178 187 15 1.1 887.5 1.0X
Deserialization 530 558 32 0.4 2647.5 0.3X

Compressed Serialized MapStatus sizes: 400 bytes
Compressed Serialized MapStatus sizes: 411 bytes
Compressed Serialized Broadcast MapStatus sizes: 2 MB


OpenJDK 64-Bit Server VM 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 10 blocks w/o broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 213 219 8 0.9 1065.1 1.0X
Deserialization 846 870 33 0.2 4228.6 0.3X
Serialization 167 175 7 1.2 835.7 1.0X
Deserialization 523 537 22 0.4 2616.2 0.3X

Compressed Serialized MapStatus sizes: 2 MB
Compressed Serialized Broadcast MapStatus sizes: 0 bytes
Expand All @@ -24,43 +24,43 @@ OpenJDK 64-Bit Server VM 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10 on Linux 4.15.
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 100 blocks w/ broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 624 709 167 0.3 3121.1 1.0X
Deserialization 885 908 22 0.2 4427.0 0.7X
Serialization 351 416 147 0.6 1754.4 1.0X
Deserialization 546 551 8 0.4 2727.6 0.6X

Compressed Serialized MapStatus sizes: 418 bytes
Compressed Serialized Broadcast MapStatus sizes: 14 MB
Compressed Serialized MapStatus sizes: 427 bytes
Compressed Serialized Broadcast MapStatus sizes: 13 MB


OpenJDK 64-Bit Server VM 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 100 blocks w/o broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 603 604 2 0.3 3014.9 1.0X
Deserialization 892 895 5 0.2 4458.7 0.7X
Serialization 320 321 1 0.6 1598.0 1.0X
Deserialization 542 549 7 0.4 2709.0 0.6X

Compressed Serialized MapStatus sizes: 14 MB
Compressed Serialized MapStatus sizes: 13 MB
Compressed Serialized Broadcast MapStatus sizes: 0 bytes


OpenJDK 64-Bit Server VM 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 1000 blocks w/ broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 4612 4945 471 0.0 23061.0 1.0X
Deserialization 1493 1495 2 0.1 7466.3 3.1X
Serialization 1671 1877 290 0.1 8357.3 1.0X
Deserialization 943 970 32 0.2 4715.8 1.8X

Compressed Serialized MapStatus sizes: 546 bytes
Compressed Serialized Broadcast MapStatus sizes: 123 MB
Compressed Serialized MapStatus sizes: 556 bytes
Compressed Serialized Broadcast MapStatus sizes: 121 MB


OpenJDK 64-Bit Server VM 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10 on Linux 4.15.0-1044-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
200000 MapOutputs, 1000 blocks w/o broadcast: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Serialization 4452 4595 202 0.0 22261.4 1.0X
Deserialization 1464 1477 18 0.1 7321.4 3.0X
Serialization 1373 1436 89 0.1 6865.0 1.0X
Deserialization 940 970 37 0.2 4699.1 1.5X

Compressed Serialized MapStatus sizes: 123 MB
Compressed Serialized MapStatus sizes: 121 MB
Compressed Serialized Broadcast MapStatus sizes: 0 bytes


Loading