[SPARK-11781][SPARKR] SparkR has problem in inferring type of raw type.

[sun-rui]

No description provided.

[shivaram]

Could you describe the problem a bit more ? Was it that raw vectors were being treated as lists ? Just curious how raw vectors differ from int vectors etc.

[SparkQA]

Test build #46148 has finished for PR 9769 at commit 07a0f33.

• This patch fails R style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sun-rui]

@shivaram, The R raw type is intended to hold raw bytes. while int vector is to hold 32-bit integer values. The R raw type maps to Spark SQL binary type, which is internally represented in Array[Byte].

This PR solves two problems:

1. Inferring of raw type is incorrect.
   > SparkR:::infer_type(as.raw(c(1, 2 ,3))) [1] "array<binary>"
   This is not correct, it should be "binary".
2. Collecting a DataFrame fails if there is any column of binary type. The bug lies in the logic that determines whether a collected column can be coerced into a atomic vector or not.

[SparkQA]

Test build #46159 has finished for PR 9769 at commit df9bca7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[shivaram]

Ok thanks for the clarification. It might take me a couple of days to get to this as the change looks a bit involved.

cc @felixcheung

[felixcheung]

should this be returned from org.apache.spark.sql.api.r.SQLUtils.dfToCols for more optimal processing? this seems like potentially a lot of data to go through

[sun-rui]

@felixcheung, you concern is reasonable. I refactor the code by using schema to determine if a collected column can be coerced into an atomic vector.

[SparkQA]

Test build #46532 has finished for PR 9769 at commit 2af83b7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sun-rui]

@shivaram, @felixcheung, could you take more look? another PR depends on this one.

[felixcheung]

nit: no space for func call: stopifnot(class(vec) != "list")

[sun-rui]

fixed

[felixcheung]

looks good.

[SparkQA]

Test build #46792 has finished for PR 9769 at commit f073c3a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[shivaram]

Could you add a comment here as well ? Something like NOTE: "binary" columns behave like complex types

[sun-rui]

added

[shivaram]

LGTM but for a minor comment.

[SparkQA]

Test build #46847 has finished for PR 9769 at commit 0b1af63.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[shivaram]

LGTM. Merging this