Skip to content

Commit 1383bd4

Browse files
yaooqinndongjoon-hyun
authored andcommitted
[SPARK-30970][K8S][CORE] Fix NPE while resolving k8s master url
### What changes were proposed in this pull request? ``` bin/spark-sql --master k8s:///https://kubernetes.docker.internal:6443 --conf spark.kubernetes.container.image=yaooqinn/spark:v2.4.4 Exception in thread "main" java.lang.NullPointerException at org.apache.spark.util.Utils$.checkAndGetK8sMasterUrl(Utils.scala:2739) at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:261) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:774) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ``` Althrough `k8s:///https://kubernetes.docker.internal:6443` is a wrong master url but should not throw npe The `case null` will never be touched. https://github.com/apache/spark/blob/3f4060c340d6bac412e8819c4388ccba226efcf3/core/src/main/scala/org/apache/spark/util/Utils.scala#L2772-L2776 ### Why are the changes needed? bug fix ### Does this PR introduce any user-facing change? no ### How was this patch tested? add ut case Closes apache#27721 from yaooqinn/SPARK-30970. Authored-by: Kent Yao <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent c0d4cc3 commit 1383bd4

File tree

2 files changed

+10
-9
lines changed

2 files changed

+10
-9
lines changed

core/src/main/scala/org/apache/spark/util/Utils.scala

Lines changed: 6 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2772,19 +2772,16 @@ private[spark] object Utils extends Logging {
27722772
}
27732773

27742774
val masterScheme = new URI(masterWithoutK8sPrefix).getScheme
2775-
val resolvedURL = masterScheme.toLowerCase(Locale.ROOT) match {
2776-
case "https" =>
2775+
2776+
val resolvedURL = Option(masterScheme).map(_.toLowerCase(Locale.ROOT)) match {
2777+
case Some("https") =>
27772778
masterWithoutK8sPrefix
2778-
case "http" =>
2779+
case Some("http") =>
27792780
logWarning("Kubernetes master URL uses HTTP instead of HTTPS.")
27802781
masterWithoutK8sPrefix
2781-
case null =>
2782-
val resolvedURL = s"https://$masterWithoutK8sPrefix"
2783-
logInfo("No scheme specified for kubernetes master URL, so defaulting to https. Resolved " +
2784-
s"URL is $resolvedURL.")
2785-
resolvedURL
27862782
case _ =>
2787-
throw new IllegalArgumentException("Invalid Kubernetes master scheme: " + masterScheme)
2783+
throw new IllegalArgumentException("Invalid Kubernetes master scheme: " + masterScheme
2784+
+ " found in URL: " + masterWithoutK8sPrefix)
27882785
}
27892786

27902787
s"k8s://$resolvedURL"

core/src/test/scala/org/apache/spark/util/UtilsSuite.scala

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1243,6 +1243,10 @@ class UtilsSuite extends SparkFunSuite with ResetSystemProperties with Logging {
12431243
intercept[IllegalArgumentException] {
12441244
Utils.checkAndGetK8sMasterUrl("k8s://foo://host:port")
12451245
}
1246+
1247+
intercept[IllegalArgumentException] {
1248+
Utils.checkAndGetK8sMasterUrl("k8s:///https://host:port")
1249+
}
12461250
}
12471251

12481252
test("stringHalfWidth") {

0 commit comments

Comments
 (0)