[SPARK-33511][SQL] Respect case sensitivity while resolving V2 partition specs

[MaxGekk]

What changes were proposed in this pull request?

1. Pre-process partition specs in ResolvePartitionSpec, and convert partition names according to the partition schema and the SQL config spark.sql.caseSensitive. In the PR, I propose to invoke normalizePartitionSpec for that. The function is used in DSv1 commands, so, the behavior will be similar to DSv1.
2. Move normalizePartitionSpec() from sql/core/.../datasources/PartitioningUtils to sql/catalyst/.../util/PartitioningUtils to use it in Catalyst's rule ResolvePartitionSpec

Why are the changes needed?

DSv1 commands like ALTER TABLE .. ADD PARTITION and ALTER TABLE .. DROP PARTITION respect the SQL config spark.sql.caseSensitive while resolving partition specs. For example:

spark-sql> CREATE TABLE tbl1 (id bigint, data string) USING parquet PARTITIONED BY (id);
spark-sql> ALTER TABLE tbl1 ADD PARTITION (ID=1);
spark-sql> SHOW PARTITIONS tbl1;
id=1

The same command fails on V2 Table catalog with error:

AnalysisException: Partition key ID not exists

Does this PR introduce any user-facing change?

Yes. After the changes, partition spec resolution works as for DSv1 (without the exception showed above).

How was this patch tested?

By running AlterTablePartitionV2SQLSuite.

[SparkQA]

Test build #131479 has finished for PR 30454 at commit 7269739.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #131480 has started for PR 30454 at commit 046f0d1.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36085/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36085/

[dongjoon-hyun]

Thank you, @MaxGekk . According to JIRA, this is only for 3.1.0, right?

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36086/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36086/

[MaxGekk]

According to JIRA, this is only for 3.1.0, right?

@dongjoon-hyun Yes, 3.0 doesn't support V2 ALTER TABLE ... PARTITION and doesn't have codes for V2 partition specs resolution.

[MaxGekk]

@HyukjinKwon @cloud-fan Could you look at this PR, please.

[cloud-fan]

nit: private[sql]

[cloud-fan]

GA passed, merging to master!