Skip to content

[HWORKS-1405] Get correct hive version in spark #40

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jul 4, 2024
Merged

[HWORKS-1405] Get correct hive version in spark #40

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ac21967
Get correct hive version in spark
gibchikafa Jul 3, 2024
3826e49
Improvement
gibchikafa Jul 3, 2024
f4c7457
Put correct hive version
gibchikafa Jul 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,8 +129,8 @@
<hive.group>io.hops.hive</hive.group>
<hive.classifier>core</hive.classifier>
<!-- Version used in Maven Hive dependency -->
<hive.version>3.0.0.13.6</hive.version>
<hive23.version>3.0.0.13.6</hive23.version>
<hive.version>3.0.0.13.5</hive.version>
<hive23.version>3.0.0.13.5</hive23.version>
<!-- Version used for internal directory structure -->
<hive.version.short>3.0</hive.version.short>
<!-- note that this should be compatible with Kafka brokers version 0.10 and up -->
Expand Down
37 changes: 20 additions & 17 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,23 +89,26 @@ private[hive] object IsolatedClientLoader extends Logging {
}

def hiveVersion(version: String): HiveVersion = {
// Fabio: Remove the last digit of the version string as it's the Hopsworks specific version
version.substring(0, version.lastIndexOf(".")) match {
case "12" | "0.12" | "0.12.0" => hive.v12
case "13" | "0.13" | "0.13.0" | "0.13.1" => hive.v13
case "14" | "0.14" | "0.14.0" => hive.v14
case "1.0" | "1.0.0" | "1.0.1" => hive.v1_0
case "1.1" | "1.1.0" | "1.1.1" => hive.v1_1
case "1.2" | "1.2.0" | "1.2.1" | "1.2.2" => hive.v1_2
case "2.0" | "2.0.0" | "2.0.1" => hive.v2_0
case "2.1" | "2.1.0" | "2.1.1" => hive.v2_1
case "2.2" | "2.2.0" => hive.v2_2
case "2.3" | "2.3.0" | "2.3.1" | "2.3.2" | "2.3.3" | "2.3.4" | "2.3.5" | "2.3.6" | "2.3.7" =>
hive.v2_3
case "3.0" | "3.0.0" => hive.v3_0
case "3.1" | "3.1.0" | "3.1.1" | "3.1.2" => hive.v3_1
case version =>
throw new UnsupportedOperationException(s"Unsupported Hive Metastore version ($version). " +
def extractMajorMinorVersion(version: String): String = {
val parts = version.split("\\.")
if (parts.length >= 2) parts(0) + "." + parts(1) else parts(0)
}

val majorMinorVersion = extractMajorMinorVersion(version)
majorMinorVersion match {
case "0.12" => hive.v12
case "0.13" => hive.v13
case "0.14" => hive.v14
case "1.0" => hive.v1_0
case "1.1" => hive.v1_1
case "1.2" => hive.v1_2
case "2.0" => hive.v2_0
case "2.1" => hive.v2_1
case "2.2" => hive.v2_2
case "2.3" => hive.v2_3
case "3.0" => hive.v3_0
case "3.1" => hive.v3_1
case _ => throw new UnsupportedOperationException(s"Unsupported Hive Metastore version ($version). " +
s"Please set ${HiveUtils.HIVE_METASTORE_VERSION.key} with a valid version.")
}
}
Expand Down