[SPARK-49163][SQL] Attempt to create table based on broken parquet partition data results should return user-facing error

[nikolamand-db]

What changes were proposed in this pull request?

Create an example parquet table with partitions and insert data in Spark:

create table t(col1 string, col2 string, col3 string) using parquet location 'some/path/parquet-test' partitioned by (col1, col2);
insert into t (col1, col2, col3) values ('a', 'b', 'c');

Go into the parquet-test path in the filesystem and try to copy parquet data file from path col1=a/col2=b directory into col1=a. After that, try to create new table based on parquet data in Spark:

create table broken_table using parquet location 'some/path/parquet-test';

This query errors with internal error. Stack trace excerpts:

org.apache.spark.SparkException: [INTERNAL_ERROR] Eagerly executed command failed. You hit a bug in Spark or the Spark plugins you use. Please, report this bug to the corresponding communities or vendors, and provide the full stack trace. SQLSTATE: XX000
...
Caused by: java.lang.AssertionError: assertion failed: Conflicting partition column names detected:        Partition column name list #0: col1
        Partition column name list #1: col1, col2For partitioned table directories, data files should only live in leaf directories.
And directories at the same level should have the same partition column name.
Please check the following directories for unexpected files or inconsistent partition column names:        file:some/path/parquet-test/col1=a
        file:some/path/parquet-test/col1=a/col2=b
  at scala.Predef$.assert(Predef.scala:279)
  at org.apache.spark.sql.execution.datasources.PartitioningUtils$.resolvePartitions(PartitioningUtils.scala:391)
...

Fix this by changing internal error to user-facing error.

Why are the changes needed?

Replace internal error with user-facing one for valid sequence of Spark SQL operations.

Does this PR introduce any user-facing change?

Yes, it presents the user with regular error instead of internal error.

How was this patch tested?

Added checks to ParquetPartitionDiscoverySuite which simulate the described scenario by manually breaking parquet table in the filesystem.

Was this patch authored or co-authored using generative AI tooling?

No.

[cloud-fan]

thanks, merging to master!

[dongjoon-hyun]

+1, LGTM. Thank you!