[SPARK-9404][SPARK-9542][SQL] unsafe array data and map data #7752

cloud-fan · 2015-07-29T17:14:06Z

This PR adds a UnsafeArrayData, current we encode it in this way:

first 4 bytes is the # elements
then each 4 byte is the start offset of the element, unless it is negative, in which case the element is null.
followed by the elements themselves

an example: [10, 11, 12, 13, null, 14] will be encoded as:
5, 28, 32, 36, 40, -44, 44, 10, 11, 12, 13, 14

Note that, when we read a UnsafeArrayData from bytes, we can read the first 4 bytes as numElements and take the rest(first 4 bytes skipped) as value region.

unsafe map data just use 2 unsafe array data, first 4 bytes is # of elements, second 4 bytes is numBytes of key array, the follows key array data and value array data.

SparkQA · 2015-07-29T17:53:56Z

Test build #38855 has finished for PR 7752 at commit bb1fc15.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class UnsafeArrayData extends ArrayData
- public static class ArrayWriter
- abstract class ArrayData extends SpecializedGetters with Serializable
- class GenericArrayData(array: Array[Any]) extends ArrayData

rxin · 2015-07-29T18:44:40Z

sql/catalyst/src/main/scala/org/apache/spark/sql/types/ArrayData.scala

this will fail on all places that use toArray in UnsafeArrayData right?

we'll never call toArray in codegen.

OK the thing I'm worried about is fallback mode, and also I want to make it so unsaferow can be passed onto normal non-codegen/unsafe operators.

SparkQA · 2015-07-30T04:10:36Z

Test build #38969 has finished for PR 7752 at commit 8e33afa.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class UnsafeArrayData extends ArrayData
- public static class ArrayWriter
- abstract class ArrayData extends SpecializedGetters with Serializable
- class GenericArrayData(array: Array[Any]) extends ArrayData

SparkQA · 2015-07-30T05:02:42Z

Test build #38984 has finished for PR 7752 at commit d589bcb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2015-07-30T17:06:46Z

I've merged the other ArrayData patch. Can you bring this one up to date? Thanks.

cloud-fan · 2015-08-01T15:14:15Z

sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeWriters.java

This is a more abstract version of UnsafeRowWriters, we can make it work for both row and array in future refactor.

SparkQA · 2015-08-01T18:39:56Z

Test build #39356 has finished for PR 7752 at commit 0bc44a4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2015-08-02T00:58:32Z

sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java

we should definitely document the binary format of UnsafeArrayData in the javadoc.

SparkQA · 2015-08-02T04:02:56Z

Test build #39400 has finished for PR 7752 at commit ba75fb7.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class UnsafeArrayData extends ArrayData
- public static class ArrayWriter
- public class UnsafeWriters
- public static class DecimalWriter
- public static class UTF8StringWriter
- public static class BinaryWriter
- public static class StructWriter
- public static class IntervalWriter
- public static class ArrayWriter
- case class FromUnsafe(child: Expression) extends UnaryExpression

SparkQA · 2015-08-02T04:39:46Z

Test build #39402 has finished for PR 7752 at commit 2d6af4b.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class UnsafeArrayData extends ArrayData
- public static class ArrayWriter
- public class UnsafeWriters
- public static class DecimalWriter
- public static class UTF8StringWriter
- public static class BinaryWriter
- public static class StructWriter
- public static class IntervalWriter
- public static class ArrayWriter
- case class FromUnsafe(child: Expression) extends UnaryExpression

davies · 2015-08-02T06:11:29Z

sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java

We may could just use UnsafeRow for ArrayType, it's not the most efficient one, but easy to catch the deadline, we could optimize it later, until we have more clear mind on it.

He's already done with it hasn't he?

SparkQA · 2015-08-02T08:44:49Z

Test build #39410 has finished for PR 7752 at commit d56708e.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
- public class UnsafeArrayData extends ArrayData
- public static class ArrayWriter
- public class UnsafeWriters
- public static class DecimalWriter
- public static class UTF8StringWriter
- public static class BinaryWriter
- public static class StructWriter
- public static class IntervalWriter
- public static class ArrayWriter
- case class FromUnsafe(child: Expression) extends UnaryExpression

SparkQA · 2015-08-02T09:06:51Z

Test build #39411 has finished for PR 7752 at commit 20d1039.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class UnsafeArrayData extends ArrayData
- public static class ArrayWriter
- public class UnsafeWriters
- public static class DecimalWriter
- public static class UTF8StringWriter
- public static class BinaryWriter
- public static class StructWriter
- public static class IntervalWriter
- public static class ArrayWriter
- case class FromUnsafe(child: Expression) extends UnaryExpression
- class SpecificSafeProjection extends $
- case class FromUTCTimestamp(left: Expression, right: Expression)
- case class ToUTCTimestamp(left: Expression, right: Expression)
- case class DateDiff(endDate: Expression, startDate: Expression)
- case class InitCap(child: Expression) extends UnaryExpression with ImplicitCastInputTypes

SparkQA · 2015-08-02T18:11:57Z

Test build #39430 has finished for PR 7752 at commit 6445289.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class UnsafeArrayData extends ArrayData
- public class UnsafeMapData extends MapData
- public class UnsafeReaders
- public static class ArrayWriter
- public static class MapWriter
- public class UnsafeWriters
- public static class DecimalWriter
- public static class UTF8StringWriter
- public static class BinaryWriter
- public static class StructWriter
- public static class IntervalWriter
- public static class ArrayWriter
- public static class MapWriter
- case class FromUnsafe(child: Expression) extends UnaryExpression

SparkQA · 2015-08-02T20:07:05Z

Test build #39436 has finished for PR 7752 at commit 36542b7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class UnsafeArrayData extends ArrayData
- public class UnsafeMapData extends MapData
- public class UnsafeReaders
- public static class ArrayWriter
- public static class MapWriter
- public class UnsafeWriters
- public static class DecimalWriter
- public static class UTF8StringWriter
- public static class BinaryWriter
- public static class StructWriter
- public static class IntervalWriter
- public static class ArrayWriter
- public static class MapWriter
- case class FromUnsafe(child: Expression) extends UnaryExpression

rxin · 2015-08-03T01:08:04Z

sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java

rxin · 2015-08-03T01:17:49Z

sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java

label baseObject as @nullable

SparkQA · 2015-08-03T03:14:43Z

Test build #39486 has finished for PR 7752 at commit 345c064.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class UnsafeArrayData extends ArrayData
- public class UnsafeMapData extends MapData
- public class UnsafeReaders
- public static class ArrayWriter
- public static class MapWriter
- public class UnsafeWriters
- public static class DecimalWriter
- public static class UTF8StringWriter
- public static class BinaryWriter
- public static class StructWriter
- public static class IntervalWriter
- public static class ArrayWriter
- public static class MapWriter
- case class FromUnsafe(child: Expression) extends UnaryExpression

SparkQA · 2015-08-03T05:14:52Z

Test build #39489 timed out for PR 7752 at commit 3269bd7 after a configured wait of 175m.

SparkQA · 2015-08-03T06:35:14Z

Test build #1303 has finished for PR 7752 at commit 3269bd7.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class UnsafeArrayData extends ArrayData
- public class UnsafeMapData extends MapData
- public class UnsafeReaders
- public static class ArrayWriter
- public static class MapWriter
- public class UnsafeWriters
- public static class DecimalWriter
- public static class UTF8StringWriter
- public static class BinaryWriter
- public static class StructWriter
- public static class IntervalWriter
- public static class ArrayWriter
- public static class MapWriter
- case class FromUnsafe(child: Expression) extends UnaryExpression

rxin · 2015-08-03T06:40:52Z

OK going to merge this.

…enerateSafe In #7752 we added `FromUnsafe` to convert nexted unsafe data like array/map/struct to safe versions. It's a quick solution and we already have `GenerateSafe` to do the conversion which is codegened. So we should remove `FromUnsafe` and implement its codegen version in `GenerateSafe`. Author: Wenchen Fan <[email protected]> Closes #8029 from cloud-fan/from-unsafe and squashes the following commits: ed40d8f [Wenchen Fan] add the copy back a93fd4b [Wenchen Fan] cogengen FromUnsafe (cherry picked from commit 106c078) Signed-off-by: Davies Liu <[email protected]>

…enerateSafe In #7752 we added `FromUnsafe` to convert nexted unsafe data like array/map/struct to safe versions. It's a quick solution and we already have `GenerateSafe` to do the conversion which is codegened. So we should remove `FromUnsafe` and implement its codegen version in `GenerateSafe`. Author: Wenchen Fan <[email protected]> Closes #8029 from cloud-fan/from-unsafe and squashes the following commits: ed40d8f [Wenchen Fan] add the copy back a93fd4b [Wenchen Fan] cogengen FromUnsafe

…enerateSafe In apache#7752 we added `FromUnsafe` to convert nexted unsafe data like array/map/struct to safe versions. It's a quick solution and we already have `GenerateSafe` to do the conversion which is codegened. So we should remove `FromUnsafe` and implement its codegen version in `GenerateSafe`. Author: Wenchen Fan <[email protected]> Closes apache#8029 from cloud-fan/from-unsafe and squashes the following commits: ed40d8f [Wenchen Fan] add the copy back a93fd4b [Wenchen Fan] cogengen FromUnsafe

…enerateSafe In apache/spark#7752 we added `FromUnsafe` to convert nexted unsafe data like array/map/struct to safe versions. It's a quick solution and we already have `GenerateSafe` to do the conversion which is codegened. So we should remove `FromUnsafe` and implement its codegen version in `GenerateSafe`. Author: Wenchen Fan <[email protected]> Closes #8029 from cloud-fan/from-unsafe and squashes the following commits: ed40d8f [Wenchen Fan] add the copy back a93fd4b [Wenchen Fan] cogengen FromUnsafe

rxin reviewed Jul 29, 2015
View reviewed changes

cloud-fan force-pushed the unsafe-array branch from bb1fc15 to 8e33afa Compare July 30, 2015 03:34

cloud-fan changed the title ~~[SPARK-9404][SQL][WIP] unsafe array data~~ [SPARK-9404][SQL] unsafe array data Jul 30, 2015

cloud-fan force-pushed the unsafe-array branch from d589bcb to 0bc44a4 Compare August 1, 2015 15:01

cloud-fan reviewed Aug 1, 2015
View reviewed changes

rxin reviewed Aug 2, 2015
View reviewed changes

cloud-fan force-pushed the unsafe-array branch from ba75fb7 to 2d6af4b Compare August 2, 2015 04:32

davies reviewed Aug 2, 2015
View reviewed changes

cloud-fan force-pushed the unsafe-array branch from 2d6af4b to d56708e Compare August 2, 2015 06:48

cloud-fan added 2 commits August 2, 2015 14:55

add unsafe array

821b8db

add comments and unsafe converter

20d1039

cloud-fan force-pushed the unsafe-array branch from d56708e to 20d1039 Compare August 2, 2015 07:07

cloud-fan added 2 commits August 2, 2015 22:37

add unsafe map

49adf26

add unit tests

6445289

cloud-fan changed the title ~~[SPARK-9404][SQL] unsafe array data~~ [SPARK-9404][SPARK-9542][SQL] unsafe array data and map data Aug 2, 2015

rxin reviewed Aug 3, 2015
View reviewed changes

sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java Outdated

Copy link

Contributor

rxin Aug 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ghe -> the

cloud-fan force-pushed the unsafe-array branch 2 times, most recently from feda148 to 345c064 Compare August 3, 2015 01:16

rxin reviewed Aug 3, 2015
View reviewed changes

sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java

Copy link

Contributor

rxin Aug 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

label baseObject as @nullable

cloud-fan force-pushed the unsafe-array branch from 345c064 to 3269bd7 Compare August 3, 2015 02:15

fix a bug

3269bd7

asfgit closed this in 608353c Aug 3, 2015

cloud-fan mentioned this pull request Aug 8, 2015

[SPARK-9738][SQL] remove FromUnsafe and add its codegen version to GenerateSafe #8029

Closed

[SPARK-9404][SPARK-9542][SQL] unsafe array data and map data #7752

[SPARK-9404][SPARK-9542][SQL] unsafe array data and map data #7752

Uh oh!

Conversation

cloud-fan commented Jul 29, 2015

Uh oh!

SparkQA commented Jul 29, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!

rxin commented Jul 30, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 1, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 2, 2015

Uh oh!

SparkQA commented Aug 2, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 2, 2015

Uh oh!

SparkQA commented Aug 2, 2015

Uh oh!

SparkQA commented Aug 2, 2015

Uh oh!

SparkQA commented Aug 2, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 3, 2015

Uh oh!

SparkQA commented Aug 3, 2015

Uh oh!

SparkQA commented Aug 3, 2015

Uh oh!

rxin commented Aug 3, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants