Skip to content

[SPARK-29175][SQL] Make additional remote maven repository in IsolatedClientLoader configurable #25849

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 7 commits into from
Closed

[SPARK-29175][SQL] Make additional remote maven repository in IsolatedClientLoader configurable #25849

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e5d189c
Add config DEFAULT_CENTRAL_REPOSITORY
xuanyuanking Sep 19, 2019
47f8402
move config to SQLConf
xuanyuanking Sep 19, 2019
a1f99cc
rename and set the config in hive tests
xuanyuanking Sep 20, 2019
a1502c5
fix comment
xuanyuanking Sep 24, 2019
7cc0607
Address comments
xuanyuanking Sep 25, 2019
5bd630c
address comment
xuanyuanking Sep 25, 2019
6e6b87c
fix doc
xuanyuanking Sep 26, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1979,6 +1979,15 @@ object SQLConf {
.doc("When true, the ArrayExists will follow the three-valued boolean logic.")
.booleanConf
.createWithDefault(true)

val ADDITIONAL_REMOTE_REPOSITORIES =
buildConf("spark.sql.additionalRemoteRepositories")
.doc("A comma-delimited string config of the optional additional remote Maven mirror " +
"repositories. This is only used for downloading Hive jars in IsolatedClientLoader " +
"if the default Maven Central repo is unreachable.")
.stringConf
.createWithDefault(
"https://maven-central.storage-download.googleapis.com/repos/central/data/")
}

/**
Expand Down
12 changes: 8 additions & 4 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ import org.apache.spark.internal.Logging
import org.apache.spark.sql.catalyst.util.quietly
import org.apache.spark.sql.hive.HiveUtils
import org.apache.spark.sql.internal.NonClosableMutableURLClassLoader
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.util.{MutableURLClassLoader, Utils}

/** Factory for `IsolatedClientLoader` with specific versions of hive. */
Expand All @@ -60,9 +61,10 @@ private[hive] object IsolatedClientLoader extends Logging {
val files = if (resolvedVersions.contains((resolvedVersion, hadoopVersion))) {
resolvedVersions((resolvedVersion, hadoopVersion))
} else {
val remoteRepos = sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES)
val (downloadedFiles, actualHadoopVersion) =
try {
(downloadVersion(resolvedVersion, hadoopVersion, ivyPath), hadoopVersion)
(downloadVersion(resolvedVersion, hadoopVersion, ivyPath, remoteRepos), hadoopVersion)
} catch {
case e: RuntimeException if e.getMessage.contains("hadoop") =>
// If the error message contains hadoop, it is probably because the hadoop
Expand All @@ -74,7 +76,8 @@ private[hive] object IsolatedClientLoader extends Logging {
"It is recommended to set jars used by Hive metastore client through " +
"spark.sql.hive.metastore.jars in the production environment.")
_sharesHadoopClasses = false
(downloadVersion(resolvedVersion, fallbackVersion, ivyPath), fallbackVersion)
(downloadVersion(
resolvedVersion, fallbackVersion, ivyPath, remoteRepos), fallbackVersion)
}
resolvedVersions.put((resolvedVersion, actualHadoopVersion), downloadedFiles)
resolvedVersions((resolvedVersion, actualHadoopVersion))
Expand Down Expand Up @@ -112,7 +115,8 @@ private[hive] object IsolatedClientLoader extends Logging {
private def downloadVersion(
version: HiveVersion,
hadoopVersion: String,
ivyPath: Option[String]): Seq[URL] = {
ivyPath: Option[String],
remoteRepos: String): Seq[URL] = {
val hiveArtifacts = version.extraDeps ++
Seq("hive-metastore", "hive-exec", "hive-common", "hive-serde")
.map(a => s"org.apache.hive:$a:${version.fullVersion}") ++
Expand All @@ -123,7 +127,7 @@ private[hive] object IsolatedClientLoader extends Logging {
SparkSubmitUtils.resolveMavenCoordinates(
hiveArtifacts.mkString(","),
SparkSubmitUtils.buildIvySettings(
Some("https://maven-central.storage-download.googleapis.com/repos/central/data/"),
Some(remoteRepos),
ivyPath),
exclusions = version.exclusions)
}
Expand Down