[SPARK-42164][CORE] Register partitioned-table-related classes to KryoSerializer

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to register partitioned-table-related classes to KryoSerializer.
Specifically, CREATE TABLE and MSCK REPAIR TABLE uses this classes.

Why are the changes needed?

To support partitioned-tables more easily with KryoSerializer. Previously, it fails like the following.

java.lang.IllegalArgumentException: Class is not registered:
org.apache.spark.util.HadoopFSUtils$SerializableBlockLocation

java.lang.IllegalArgumentException: Class is not registered:
org.apache.spark.util.HadoopFSUtils$SerializableFileStatus

java.lang.IllegalArgumentException: Class is not registered:
org.apache.spark.sql.execution.command.PartitionStatistics

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs and manually tests.

TEST TABLE

$ tree /tmp/t
/tmp/t
├── p=1
│   └── users.orc
├── p=10
│   └── users.orc
├── p=11
│   └── users.orc
├── p=2
│   └── users.orc
├── p=3
│   └── users.orc
├── p=4
│   └── users.orc
├── p=5
│   └── users.orc
├── p=6
│   └── users.orc
├── p=7
│   └── users.orc
├── p=8
│   └── users.orc
└── p=9
    └── users.orc

CREATE PARTITIONED TABLES AND RECOVER PARTITIONS

$ bin/spark-shell -c spark.kryo.registrationRequired=true -c spark.serializer=org.apache.spark.serializer.KryoSerializer -c spark.sql.sources.parallelPartitionDiscovery.threshold=1

scala> sql("CREATE TABLE t USING ORC LOCATION '/tmp/t'").show()
++                                                                              
||
++
++


scala> sql("MSCK REPAIR TABLE t").show()
++
||
++
++

[dongjoon-hyun]

Since these are private case class of HadoopFSUtils, we need to register like this.

[dongjoon-hyun]

Could you review this, @viirya ?

[dongjoon-hyun]

Thank you, @viirya ! Merged to master.