[SPARK-26709][SQL][BRANCH-2.3] OptimizeMetadataOnlyQuery does not handle empty records correctly

[gengliangwang]

What changes were proposed in this pull request?

When reading from empty tables, the optimization OptimizeMetadataOnlyQuery may return wrong results:

sql("CREATE TABLE t (col1 INT, p1 INT) USING PARQUET PARTITIONED BY (p1)")
sql("INSERT INTO TABLE t PARTITION (p1 = 5) SELECT ID FROM range(1, 1)")
sql("SELECT MAX(p1) FROM t")

The result is supposed to be null. However, with the optimization the result is 5.

The rule is originally ported from https://issues.apache.org/jira/browse/HIVE-1003 in #13494. In Hive, the rule is disabled by default in a later release(https://issues.apache.org/jira/browse/HIVE-15397), due to the same problem.

It is hard to completely avoid the correctness issue. Because data sources like Parquet can be metadata-only. Spark can't tell whether it is empty or not without actually reading it. This PR disable the optimization by default.

How was this patch tested?

Unit test

[gengliangwang]

This PR is to port #23635 to branch 2.3

[gatorsmile]

LGTM pending Jenkins.

[SparkQA]

Test build #101667 has finished for PR 23648 at commit d782e4a.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[gengliangwang]

retest this please.

[SparkQA]

Test build #101670 has finished for PR 23648 at commit d782e4a.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #101677 has finished for PR 23648 at commit deb84ea.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[maropu]

It seems the test couldn't create an empty parition crrectly..... how about this?

sql("CREATE TABLE t (col1 INT, p1 INT) USING PARQUET PARTITIONED BY (p1)")
sql("INSERT INTO TABLE t PARTITION (p1 = 5) SELECT 1")
sql("TRUNCATE TABLE t PARTITION (p1 = 5)")

[gengliangwang]

@maropu Thanks.

sql("TRUNCATE TABLE t PARTITION (p1 = 5)")

will throw exception

org.apache.spark.sql.catalyst.analysis.NoSuchPartitionException: Partition not found in table 't' database 'default':
[info] p1 -> 5;

[SparkQA]

Test build #101683 has finished for PR 23648 at commit 096bb6c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #101682 has finished for PR 23648 at commit bf280cf.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gengliangwang]

retest this please.

[SparkQA]

Test build #101688 has finished for PR 23648 at commit 096bb6c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[maropu]

LGTM

[maropu]

Thanks! Merged to branch-2.3.