[SPARK-27291][SQL] PartitioningAwareFileIndex: Filter out empty files on listing files

[gengliangwang]

What changes were proposed in this pull request?

In #23130, all empty files are excluded from target file splits in FileSourceScanExec.
In File source V2, we should keep the same behavior.

This PR suggests to filter out empty files on listing files in PartitioningAwareFileIndex so that the upper level doesn't need to handle them.

How was this patch tested?

Unit test

[gengliangwang]

@cloud-fan @HyukjinKwon @dongjoon-hyun

[MaxGekk]

Looks like the filtered out empty files can impact on calculation of maxSplitBytes:

spark/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FilePartition.scala

Line 92 in c0632ce

val totalBytes = selectedPartitions.flatMap(_.files.map(_.getLen + openCostInBytes)).sum

. Probably, we need to handle the case there or filter empty files earlier.

[SparkQA]

Test build #104005 has finished for PR 24227 at commit 664fe9b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gengliangwang]

@MaxGekk Thanks for the suggestion. I have updated the PR.
My only concern is that the metrics "numFiles" will be affected. The empty files won't be taken into account. But it should be fine since Spark doesn't read these empty files.

[MaxGekk]

LGTM

[cloud-fan]

thanks, merging to master!

[SparkQA]

Test build #104016 has finished for PR 24227 at commit 4bf5cec.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.