From 67e7e589a3eabab460dac5fcccd2d67a71cd0942 Mon Sep 17 00:00:00 2001 From: Dongjoon Hyun Date: Wed, 29 Jan 2025 00:20:54 -0800 Subject: [PATCH] [SPARK-51029][BUILD] Remove `hive-llap-common` compile dependency --- dev/deps/spark-deps-hadoop-3-hive-2.3 | 1 - docs/sql-migration-guide.md | 1 + pom.xml | 2 +- 3 files changed, 2 insertions(+), 2 deletions(-) diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3 index 96d5f9d477143..ca52760a3368b 100644 --- a/dev/deps/spark-deps-hadoop-3-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-3-hive-2.3 @@ -86,7 +86,6 @@ hive-cli/2.3.10//hive-cli-2.3.10.jar hive-common/2.3.10//hive-common-2.3.10.jar hive-exec/2.3.10/core/hive-exec-2.3.10-core.jar hive-jdbc/2.3.10//hive-jdbc-2.3.10.jar -hive-llap-common/2.3.10//hive-llap-common-2.3.10.jar hive-metastore/2.3.10//hive-metastore-2.3.10.jar hive-serde/2.3.10//hive-serde-2.3.10.jar hive-service-rpc/4.0.0//hive-service-rpc-4.0.0.jar diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md index 254c54a414a7e..f459a88d8e148 100644 --- a/docs/sql-migration-guide.md +++ b/docs/sql-migration-guide.md @@ -31,6 +31,7 @@ license: | - Since Spark 4.0, any read of SQL tables takes into consideration the SQL configs `spark.sql.files.ignoreCorruptFiles`/`spark.sql.files.ignoreMissingFiles` instead of the core config `spark.files.ignoreCorruptFiles`/`spark.files.ignoreMissingFiles`. - Since Spark 4.0, when reading SQL tables hits `org.apache.hadoop.security.AccessControlException` and `org.apache.hadoop.hdfs.BlockMissingException`, the exception will be thrown and fail the task, even if `spark.sql.files.ignoreCorruptFiles` is set to `true`. - Since Spark 4.0, `spark.sql.hive.metastore` drops the support of Hive prior to 2.0.0 as they require JDK 8 that Spark does not support anymore. Users should migrate to higher versions. +- Since Spark 4.0, Spark removes `hive-llap-common` dependency. To restore the previous behavior, add `hive-llap-common` jar to the class path. - Since Spark 4.0, `spark.sql.parquet.compression.codec` drops the support of codec name `lz4raw`, please use `lz4_raw` instead. - Since Spark 4.0, when overflowing during casting timestamp to byte/short/int under non-ansi mode, Spark will return null instead a wrapping value. - Since Spark 4.0, the `encode()` and `decode()` functions support only the following charsets 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16', 'UTF-32'. To restore the previous behavior when the function accepts charsets of the current JDK used by Spark, set `spark.sql.legacy.javaCharsets` to `true`. diff --git a/pom.xml b/pom.xml index 769dad270af77..a7c59c0203437 100644 --- a/pom.xml +++ b/pom.xml @@ -274,7 +274,7 @@ compile compile compile - compile + test compile compile compile