[SPARK-26736][SQL] Partition pruning through nondeterministic expressions in Hive tables

[maropu]

What changes were proposed in this pull request?

This PR intends to improve partition pruning for nondeterministic expressions in Hive tables:

Before this PR:

scala> sql("""create table test(id int) partitioned by (dt string)""")
scala> sql("""select * from test where dt='20190101' and rand() < 0.5""").explain()

== Physical Plan ==
*(1) Filter ((isnotnull(dt#19) AND (dt#19 = 20190101)) AND (rand(6515336563966543616) < 0.5))
+- Scan hive default.test [id#18, dt#19], HiveTableRelation `default`.`test`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [id#18], [dt#19], Statistics(sizeInBytes=8.0 EiB)

After this PR:

== Physical Plan ==
*(1) Filter (rand(-9163956883277176328) < 0.5)
+- Scan hive default.test [id#0, dt#1], HiveTableRelation `default`.`test`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [id#0], [dt#1], Statistics(sizeInBytes=8.0 EiB), [isnotnull(dt#1), (dt#1 = 20190101)]

This PR is the rework of #24118.

Why are the changes needed?

For better performance.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit tests added.

[SparkQA]

Test build #116779 has finished for PR 27219 at commit 7d8d661.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[maropu]

cc: @cloud-fan @Ngone51

[cloud-fan]

not sure we need this. As a UT, we know that it doesn't matter how predicates are ANDed together.

[maropu]

ok, I will drop.

[SparkQA]

Test build #116789 has finished for PR 27219 at commit 731e2a2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[maropu]

Merged to master.

[dongjoon-hyun]

+1, late LGTM.