[SPARK-9390][SQL] create a wrapper for array type #7724

cloud-fan · 2015-07-28T07:14:39Z

No description provided.

cloud-fan · 2015-07-28T07:16:06Z

sql/catalyst/src/main/scala/org/apache/spark/sql/types/GenericArrayData.scala

the generic get and array method is useful in interpreted mode, but it's a lot painful to handle Any between scala and java, so I put this interface in scala.

cloud-fan · 2015-07-28T07:17:32Z

cc @rxin

SparkQA · 2015-07-28T07:19:35Z

Test build #38685 has finished for PR 7724 at commit 53cf9fc.

This patch fails RAT tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
- abstract class ArrayData extends SpecializedGetters
- class GenericArrayData(val array: Array[Any]) extends ArrayData

cloud-fan · 2015-07-28T15:06:30Z

cc @rxin, I underestimated this work, It took me a lot of time to cover all corner cases...

SparkQA · 2015-07-28T15:12:12Z

Test build #38707 has finished for PR 7724 at commit 21d71c6.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class ArrayData extends SpecializedGetters with Serializable
- class GenericArrayData(array: Array[Any]) extends ArrayData

cloud-fan · 2015-07-28T15:15:07Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala

I think we should avoid using InternalRow in test directly, which means we are manually converting external type to internal type. Using Row and let converter do this work seems better.

That makes sense.

SparkQA · 2015-07-28T15:22:24Z

Test build #38708 has finished for PR 7724 at commit 7efc4c6.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class ArrayData extends SpecializedGetters with Serializable
- class GenericArrayData(array: Array[Any]) extends ArrayData

SparkQA · 2015-07-28T15:36:51Z

Test build #38710 has finished for PR 7724 at commit 7394d0a.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class ArrayData extends SpecializedGetters with Serializable
- class GenericArrayData(array: Array[Any]) extends ArrayData

SparkQA · 2015-07-28T16:52:58Z

Test build #38718 has finished for PR 7724 at commit 1694fe6.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class ArrayData extends SpecializedGetters with Serializable
- class GenericArrayData(array: Array[Any]) extends ArrayData

rxin · 2015-07-28T22:17:46Z

sql/catalyst/src/main/scala/org/apache/spark/sql/types/GenericArrayData.scala

can you move this into its own file

cloud-fan · 2015-07-29T02:22:58Z

sql/catalyst/src/main/scala/org/apache/spark/sql/types/ArrayData.scala

I think we need specialized version of toArray, and get rid of calling generic getter inside it. We can do it after adding special getter for map and removing generic getter.

SparkQA · 2015-07-29T02:24:50Z

Test build #38790 has finished for PR 7724 at commit 485e1d9.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-29T04:06:50Z

Test build #38797 has finished for PR 7724 at commit 08b6cfd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class ArrayData extends SpecializedGetters with Serializable
- class GenericArrayData(array: Array[Any]) extends ArrayData

SparkQA · 2015-07-29T06:08:14Z

Test build #38804 has finished for PR 7724 at commit b66ed9b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class ArrayData extends SpecializedGetters with Serializable
- class GenericArrayData(array: Array[Any]) extends ArrayData

SparkQA · 2015-07-29T07:54:01Z

Test build #38818 has finished for PR 7724 at commit 9952110.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class ArrayData extends SpecializedGetters with Serializable
- class GenericArrayData(array: Array[Any]) extends ArrayData

SparkQA · 2015-07-29T09:26:53Z

Test build #38824 has finished for PR 7724 at commit ab98286.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class ArrayData extends SpecializedGetters with Serializable
- class GenericArrayData(array: Array[Any]) extends ArrayData

cloud-fan · 2015-07-29T09:49:07Z

retest this please.

SparkQA · 2015-07-29T10:28:07Z

Test build #145 has finished for PR 7724 at commit ab98286.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-29T10:36:16Z

Test build #38831 has finished for PR 7724 at commit ab98286.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class ArrayData extends SpecializedGetters with Serializable
- class GenericArrayData(array: Array[Any]) extends ArrayData

SparkQA · 2015-07-29T17:28:09Z

Test build #38846 has finished for PR 7724 at commit e9f2fc6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class ArrayData extends SpecializedGetters with Serializable
- class GenericArrayData(array: Array[Any]) extends ArrayData

SparkQA · 2015-07-29T17:34:23Z

Test build #38844 has finished for PR 7724 at commit 0772ca0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-30T02:12:49Z

Test build #38931 has finished for PR 7724 at commit 3a0147d.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class ArrayData extends SpecializedGetters with Serializable
- class GenericArrayData(array: Array[Any]) extends ArrayData

cloud-fan · 2015-07-30T03:05:23Z

cc @rxin I think it's close, there is still a pyspark doc string test failue:

File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/ml/recommendation.py", line 73, in __main__.ALS
Failed example:
    model.userFactors.orderBy("id").collect()
Expected:
    [Row(id=0, features=[...]), Row(id=1, ...), Row(id=2, ...)]
Got:
    [Row(id=0, features=(None, None, None, None, None, None, None, None, None, None)), Row(id=1, features=(None, None, None, None, None, None, None, None, None, None)), Row(id=2, features=(None, None, None, None, None, None, None, None, None, None))]

Does any body have an idea on it?

One possible reason I guess is: before ArrayData, we use ArraySeq as external array value, but now we use WrappedArray, will this make difference?

rxin · 2015-07-30T03:57:16Z

cc @davies can you take a look?

rxin · 2015-07-30T04:12:40Z

Maybe this is the problem? https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUDFs.scala#L125

SparkQA · 2015-07-30T04:48:46Z

Test build #38981 has finished for PR 7724 at commit a24c0a5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class ArrayData extends SpecializedGetters with Serializable
- class GenericArrayData(array: Array[Any]) extends ArrayData

cloud-fan · 2015-07-30T04:58:13Z

retest this please.

SparkQA · 2015-07-30T05:19:40Z

Test build #153 has finished for PR 7724 at commit a24c0a5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-30T05:29:08Z

Test build #38990 has finished for PR 7724 at commit a24c0a5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class ArrayData extends SpecializedGetters with Serializable
- class GenericArrayData(array: Array[Any]) extends ArrayData

cloud-fan · 2015-07-30T06:00:59Z

retest this please.

SparkQA · 2015-07-30T07:27:46Z

Test build #157 has finished for PR 7724 at commit a24c0a5.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-30T07:31:21Z

Test build #39007 has finished for PR 7724 at commit a24c0a5.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class ArrayData extends SpecializedGetters with Serializable
- class GenericArrayData(array: Array[Any]) extends ArrayData

SparkQA · 2015-07-30T09:16:50Z

Test build #39025 has finished for PR 7724 at commit d0408a1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2015-07-30T09:43:21Z

cc @rxin, I'll specialize toArray when working on MapData.

rxin · 2015-07-30T17:04:31Z

Thanks - I've merged this.

cloud-fan reviewed Jul 28, 2015
View reviewed changes

cloud-fan force-pushed the array-data branch from 53cf9fc to 21d71c6 Compare July 28, 2015 15:04

cloud-fan changed the title ~~[SPARK-9390][SQL][WIP] create a wrapper for array type~~ [SPARK-9390][SQL] create a wrapper for array type Jul 28, 2015

cloud-fan reviewed Jul 28, 2015
View reviewed changes

cloud-fan force-pushed the array-data branch from 21d71c6 to 7efc4c6 Compare July 28, 2015 15:16

cloud-fan force-pushed the array-data branch from 7394d0a to 1694fe6 Compare July 28, 2015 16:42

rxin reviewed Jul 28, 2015
View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/types/GenericArrayData.scala Outdated

Copy link

Contributor

rxin Jul 28, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you move this into its own file

cloud-fan force-pushed the array-data branch from 1694fe6 to 485e1d9 Compare July 29, 2015 02:15

cloud-fan reviewed Jul 29, 2015
View reviewed changes

cloud-fan force-pushed the array-data branch 2 times, most recently from 0772ca0 to e9f2fc6 Compare July 29, 2015 13:27

cloud-fan mentioned this pull request Jul 29, 2015

[SPARK-9404][SPARK-9542][SQL] unsafe array data and map data #7752

Closed

cloud-fan force-pushed the array-data branch from 5dcc85f to 3a0147d Compare July 30, 2015 00:35

cloud-fan force-pushed the array-data branch from a24c0a5 to d0408a1 Compare July 30, 2015 07:34

cloud-fan added 10 commits July 30, 2015 15:34

create a wrapper for array type

7e283e2

remove sizeInBytes and add license

d4a38da

address comment

4346290

fix hive

e719afc

fix mllib

43e9816

remove element type parameter from getArray

8cb8842

fix hive again...

6dbfa6f

fix hive...

f39256c

rebase

661e608

fix python

d0408a1

asfgit closed this in c0cc0ea Jul 30, 2015

joshrosen-stripe mentioned this pull request Jan 7, 2020

[SPARK-30414][SQL] ParquetRowConverter optimizations: arrays, maps, plus misc. constant factors #27089

Closed

[SPARK-9390][SQL] create a wrapper for array type #7724

[SPARK-9390][SQL] create a wrapper for array type #7724

Uh oh!

Conversation

cloud-fan commented Jul 28, 2015

Uh oh!

cloud-fan Jul 28, 2015

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Jul 28, 2015

Uh oh!

SparkQA commented Jul 28, 2015

Uh oh!

cloud-fan commented Jul 28, 2015

Uh oh!

SparkQA commented Jul 28, 2015

Uh oh!

cloud-fan Jul 28, 2015

Choose a reason for hiding this comment

Uh oh!

rxin Jul 28, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 28, 2015

Uh oh!

SparkQA commented Jul 28, 2015

Uh oh!

SparkQA commented Jul 28, 2015

Uh oh!

rxin Jul 28, 2015

Choose a reason for hiding this comment

Uh oh!

cloud-fan Jul 29, 2015

Choose a reason for hiding this comment

Uh oh!

rxin Jul 29, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 29, 2015

Uh oh!

SparkQA commented Jul 29, 2015

Uh oh!

SparkQA commented Jul 29, 2015

Uh oh!

SparkQA commented Jul 29, 2015

Uh oh!

SparkQA commented Jul 29, 2015

Uh oh!

cloud-fan commented Jul 29, 2015

Uh oh!

SparkQA commented Jul 29, 2015

Uh oh!

SparkQA commented Jul 29, 2015

Uh oh!

SparkQA commented Jul 29, 2015

Uh oh!

SparkQA commented Jul 29, 2015

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!

cloud-fan commented Jul 30, 2015

Uh oh!

rxin commented Jul 30, 2015

Uh oh!

rxin commented Jul 30, 2015

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!

cloud-fan commented Jul 30, 2015

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!

cloud-fan commented Jul 30, 2015

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!