[SPARK-7097][SQL]: Partitioned tables should only consider referred partitions in query during size estimation for checking against autoBroadcastJoinThreshold

[saucam]

This PR attempts to add support for better size estimation in case of partitioned tables so that only the referred partition's size are taken into consideration when testing against autoBroadCastJoinThreshold and deciding whether to create a broadcast join or shuffle hash join.

We can use the values that get stored in the hive metastore during alter table / insert into partition commands to estimate the size of each of the referred partitions.

In most cases, since both alter table query and 'insert into table partition <part=val> select * from .....' store the partition size in the metastore automatically, we expect to get the correct value of partition size. We could use Analyze table query as well in case there is some mismatch.

[SparkQA]

Test build #30860 has finished for PR 5668 at commit b0beb34.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[saucam]

retest please

[SparkQA]

Test build #30868 has finished for PR 5668 at commit d25bc2a.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #30919 has finished for PR 5668 at commit b4651fd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #30981 has started for PR 5668 at commit ce89b15.

[SparkQA]

Test build #31139 has finished for PR 5668 at commit d5b4b52.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #32496 has finished for PR 5668 at commit 136b594.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #32592 has finished for PR 5668 at commit 449db10.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #36258 has finished for PR 5668 at commit 75fc3e7.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #36276 has finished for PR 5668 at commit dd630e7.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

Looks like an old patch. @yhuai would you mind taking a look?

[JoshRosen]

Hi @saucam, I'm going through the backlog of old pull requests and noticed that this PR and #6767 seem to both be trying to solve the same issue. Would you mind taking a look at that other PR to help figure out which approach we should move forward with?

[chenghao-intel]

Actually my concern on this, probably we don't want to duplicate the optimizer rules here. A better idea is to reflect the real statistic info in MetastoreRelation as @navis did in #6767, so the default optimizer will handle the rest for us.

[SparkQA]

Test build #44983 has finished for PR 5668 at commit dd630e7.

• This patch fails to build.
• This patch does not merge cleanly.
• This patch adds no public classes.