Commit 1383bd4
[SPARK-30970][K8S][CORE] Fix NPE while resolving k8s master url
### What changes were proposed in this pull request?
```
bin/spark-sql --master k8s:///https://kubernetes.docker.internal:6443 --conf spark.kubernetes.container.image=yaooqinn/spark:v2.4.4
Exception in thread "main" java.lang.NullPointerException
at org.apache.spark.util.Utils$.checkAndGetK8sMasterUrl(Utils.scala:2739)
at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:261)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:774)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```
Althrough `k8s:///https://kubernetes.docker.internal:6443` is a wrong master url but should not throw npe
The `case null` will never be touched.
https://github.com/apache/spark/blob/3f4060c340d6bac412e8819c4388ccba226efcf3/core/src/main/scala/org/apache/spark/util/Utils.scala#L2772-L2776
### Why are the changes needed?
bug fix
### Does this PR introduce any user-facing change?
no
### How was this patch tested?
add ut case
Closes apache#27721 from yaooqinn/SPARK-30970.
Authored-by: Kent Yao <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>1 parent c0d4cc3 commit 1383bd4
File tree
2 files changed
+10
-9
lines changed- core/src
- main/scala/org/apache/spark/util
- test/scala/org/apache/spark/util
2 files changed
+10
-9
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2772 | 2772 | | |
2773 | 2773 | | |
2774 | 2774 | | |
2775 | | - | |
2776 | | - | |
| 2775 | + | |
| 2776 | + | |
| 2777 | + | |
2777 | 2778 | | |
2778 | | - | |
| 2779 | + | |
2779 | 2780 | | |
2780 | 2781 | | |
2781 | | - | |
2782 | | - | |
2783 | | - | |
2784 | | - | |
2785 | | - | |
2786 | 2782 | | |
2787 | | - | |
| 2783 | + | |
| 2784 | + | |
2788 | 2785 | | |
2789 | 2786 | | |
2790 | 2787 | | |
| |||
Lines changed: 4 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1243 | 1243 | | |
1244 | 1244 | | |
1245 | 1245 | | |
| 1246 | + | |
| 1247 | + | |
| 1248 | + | |
| 1249 | + | |
1246 | 1250 | | |
1247 | 1251 | | |
1248 | 1252 | | |
| |||
0 commit comments