[SPARK-50886][BUILD][3.5] Upgrade Avro to 1.11.4

### What changes were proposed in this pull request?

This PR aims to upgrade Avro to 1.11.4.

### Why are the changes needed?

To bring the latest bug fixes like the following.
- https://github.com/apache/avro/releases/tag/release-1.11.3
    - apache/avro#2432
    - apache/avro#2318
    - apache/avro#1720
- https://github.com/apache/avro/releases/tag/release-1.11.4
    - apache/avro#2980

### Does this PR introduce _any_ user-facing change?

Yes, this is a dependency change.

### How was this patch tested?

Pass the CIs.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#49563 from dongjoon-hyun/SPARK-50886.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>

Author: dongjoon-hyun

connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala

@@ -81,14 +81,14 @@ private[sql] class AvroOptions(
 
   /**
    * Top level record name in write result, which is required in Avro spec.
-   * See https://avro.apache.org/docs/1.11.2/specification/#schema-record .
+   * See https://avro.apache.org/docs/1.11.4/specification/#schema-record .
    * Default value is "topLevelRecord"
    */
   val recordName: String = parameters.getOrElse(RECORD_NAME, "topLevelRecord")
 
   /**
    * Record namespace in write result. Default value is "".
-   * See Avro spec for details: https://avro.apache.org/docs/1.11.2/specification/#schema-record .
+   * See Avro spec for details: https://avro.apache.org/docs/1.11.4/specification/#schema-record .
    */
   val recordNamespace: String = parameters.getOrElse(RECORD_NAMESPACE, "")
 

dev/deps/spark-deps-hadoop-3-hive-2.3

@@ -21,9 +21,9 @@ arrow-memory-core/12.0.1//arrow-memory-core-12.0.1.jar
 arrow-memory-netty/12.0.1//arrow-memory-netty-12.0.1.jar
 arrow-vector/12.0.1//arrow-vector-12.0.1.jar
 audience-annotations/0.5.0//audience-annotations-0.5.0.jar
-avro-ipc/1.11.2//avro-ipc-1.11.2.jar
-avro-mapred/1.11.2//avro-mapred-1.11.2.jar
-avro/1.11.2//avro-1.11.2.jar
+avro-ipc/1.11.4//avro-ipc-1.11.4.jar
+avro-mapred/1.11.4//avro-mapred-1.11.4.jar
+avro/1.11.4//avro-1.11.4.jar
 aws-java-sdk-bundle/1.12.262//aws-java-sdk-bundle-1.12.262.jar
 azure-data-lake-store-sdk/2.3.9//azure-data-lake-store-sdk-2.3.9.jar
 azure-keyvault-core/1.0.0//azure-keyvault-core-1.0.0.jar

docs/sql-data-sources-avro.md

@@ -417,7 +417,7 @@ applications. Read the [Advanced Dependency Management](https://spark.apache
 Submission Guide for more details.
 
 ## Supported types for Avro -> Spark SQL conversion
-Currently Spark supports reading all [primitive types](https://avro.apache.org/docs/1.11.2/specification/#primitive-types) and [complex types](https://avro.apache.org/docs/1.11.2/specification/#complex-types) under records of Avro.
+Currently Spark supports reading all [primitive types](https://avro.apache.org/docs/1.11.4/specification/#primitive-types) and [complex types](https://avro.apache.org/docs/1.11.4/specification/#complex-types) under records of Avro.
 <table>
   <thead><tr><th><b>Avro type</b></th><th><b>Spark SQL type</b></th></tr></thead>
   <tr>
@@ -481,7 +481,7 @@ In addition to the types listed above, it supports reading `union` types. The fo
 3. `union(something, null)`, where something is any supported Avro type. This will be mapped to the same Spark SQL type as that of something, with nullable set to true.
 All other union types are considered complex. They will be mapped to StructType where field names are member0, member1, etc., in accordance with members of the union. This is consistent with the behavior when converting between Avro and Parquet.
 
-It also supports reading the following Avro [logical types](https://avro.apache.org/docs/1.11.2/specification/#logical-types):
+It also supports reading the following Avro [logical types](https://avro.apache.org/docs/1.11.4/specification/#logical-types):
 
 <table>
   <thead><tr><th><b>Avro logical type</b></th><th><b>Avro type</b></th><th><b>Spark SQL type</b></th></tr></thead>

pom.xml

@@ -158,7 +158,7 @@
     -->
     <codahale.metrics.version>4.2.19</codahale.metrics.version>
     <!-- Should be consistent with SparkBuild.scala and docs -->
-    <avro.version>1.11.2</avro.version>
+    <avro.version>1.11.4</avro.version>
     <aws.kinesis.client.version>1.12.0</aws.kinesis.client.version>
     <!-- Should be consistent with Kinesis client dependency -->
     <aws.java.sdk.version>1.11.655</aws.java.sdk.version>
@@ -1485,6 +1485,10 @@
         <artifactId>avro</artifactId>
         <version>${avro.version}</version>
         <exclusions>
+          <exclusion>
+            <groupId>org.apache.commons</groupId>
+            <artifactId>commons-compress</artifactId>
+          </exclusion>
           <exclusion>
             <groupId>commons-io</groupId>
             <artifactId>commons-io</artifactId>

project/SparkBuild.scala

@@ -1103,7 +1103,8 @@ object DependencyOverrides {
     dependencyOverrides += "com.google.guava" % "guava" % guavaVersion,
     dependencyOverrides += "xerces" % "xercesImpl" % "2.12.2",
     dependencyOverrides += "jline" % "jline" % "2.14.6",
-    dependencyOverrides += "org.apache.avro" % "avro" % "1.11.2")
+    dependencyOverrides += "org.apache.avro" % "avro" % "1.11.4",
+    dependencyOverrides += "org.apache.commons" % "commons-compress" % "1.23.0")
 }
 
 /**

sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala

@@ -908,7 +908,7 @@ class HiveClientSuite(version: String, allVersions: Seq[String])
   test("Decimal support of Avro Hive serde") {
     val tableName = "tab1"
     // TODO: add the other logical types. For details, see the link:
-    // https://avro.apache.org/docs/1.11.2/specification/#logical-types
+    // https://avro.apache.org/docs/1.11.4/specification/#logical-types
     val avroSchema =
     """{
       |  "name": "test_record",