[SPARK-12104][SPARKR] collect() does not handle multiple columns with same name.

[sun-rui]

No description provided.

[shivaram]

cc @falaki

[SparkQA]

Test build #47114 has finished for PR 10118 at commit b3c654f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[felixcheung]

looks good

[falaki]

This is slightly different from 1.5. We will get exact same column names in local data.frame. In Spark 1.5 subsequent instances of the same name are appended with numbers. I am not sure which one is better. In fact I slightly prefer your suggested behavior. But just in case others want to chime in: cc @shivaram

[felixcheung]

I think the current behavior in 1.6 is actually an unintentional change from a recent change in the collect() code
Matching back to 1.5.x seems to make sense

[sun-rui]

I tested with Spark 1.4.1 and 1.5.1, both just have the same names instead of making the duplicated names unique. So this PR's behavior is backward-compatible.

>df <- createDataFrame(sqlContext, list(list(1, 2)), schema = c("a", "a"))
>collect(df)
  a a
1 1 2

Actually, it is very easy to make unique column names, like:

names(df) <- make.names(names(x), unique = TRUE)

But we need discussion is this preferred behavior?

[shivaram]

Yeah lets keep the local DF names consistent with the schema in SQL. (i.e. duplicated name, name is fine). If this is a breaking change we can add a note in the release notes

[shivaram]

@falaki Just curious: what is the query you used to create the numbered columns ?

[falaki]

I was using a left outer join and then collecting it.

[shivaram]

@sun-rui Can we add a test with left-outer join and then collect ?

[falaki]

BTW: as I said, I like this behavior.

[shivaram]

Yeah this behavior is fine. I just want to make sure that example doesn't trigger some other code path etc.

[falaki]

Thanks!

[sun-rui]

@falaki, I can't reproduce your result. for example, in Spark 1.5.1:

> df <- createDataFrame(sqlContext, list(list(1,"1")), schema = c("key", "value"))
> df1 <- join(df, df)
> collect(df1)
  key value key value
1   1     1   1     1
> df1 <- join(df, df, df$key == df$key, "left_outer")
> collect(df1)
  key value key value                                                           
1   1     1   1     1
> names(collect(df1))
[1] "key"   "value" "key"   "value"

[shivaram]

@sun-rui Was your test done with the patch ? I think @falaki 's example was on 1.5.1

[sun-rui]

@shivaram, no, done with 1.5.1

[shivaram]

Ok. Change LGTM. Merging this. We can discuss the left_join issue later if required

[SparkQA]

Test build #47178 has finished for PR 10118 at commit 3760410.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.