[SPARK-11949][SQL] Check bitmasks to set nullable property

[viirya]

Following up #10038.

We can use bitmasks to determine which grouping expressions need to be set as nullable.

cc @yhuai

[viirya]

I think this should not need a JIRA. If it is needed, please let me know. Thanks.

[SparkQA]

Test build #46965 has finished for PR 10067 at commit 69be01d.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

How about we use the original jira number?

[yhuai]

c87531b#commitcomment-14712726 looks like a good idea.

[viirya]

I think we did the same thing? I can't see the difference between this and @hvanhovell's suggestion?

[hvanhovell]

It is the same thing. I only suggested to aggregate the bitmap in advance: val nonNullBitmask = x.bitmasks.reduce(_ & _) and to use the nonNullBitmask to check for (non) nullability in the loop. It should be a tiny tiny tiny bit quicker (no re-iteration required , and less one closure).

[viirya]

Thanks. That's cool. I updated this according to your suggestion.

[viirya]

retest this please.

[SparkQA]

Test build #47006 has finished for PR 10067 at commit 69be01d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #47016 has finished for PR 10067 at commit 723d24a.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

retest this please.

[SparkQA]

Test build #47035 has finished for PR 10067 at commit 723d24a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

Thanks! Merging to master and branch 1.6.

[viirya]

@yhuai Thank you.

[yhuai]

no problem!