Skip to content

Commit f9cd5cc

Browse files
SaintBacchustdas
authored andcommitted
[SPARK-8619] [STREAMING] Don't recover keytab and principal configuration within Streaming checkpoint
[Client.scala](https://github.com/apache/spark/blob/master/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L786) will change these configurations, so this would cause the problem that the Streaming recover logic can't find the local keytab file(since configuration was changed) ```scala sparkConf.set("spark.yarn.keytab", keytabFileName) sparkConf.set("spark.yarn.principal", args.principal) ``` Problem described at [Jira](https://issues.apache.org/jira/browse/SPARK-8619) Author: huangzhaowei <[email protected]> Closes #7008 from SaintBacchus/SPARK-8619 and squashes the following commits: d50dbdf [huangzhaowei] Delect one blank space 9b8e92c [huangzhaowei] Fix code style and add a short comment. 0d8f800 [huangzhaowei] Don't recover keytab and principal configuration within Streaming checkpoint. (cherry picked from commit d16a944) Signed-off-by: Tathagata Das <[email protected]>
1 parent c83ec10 commit f9cd5cc

File tree

1 file changed

+14
-2
lines changed

1 file changed

+14
-2
lines changed

streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -44,11 +44,23 @@ class Checkpoint(@transient ssc: StreamingContext, val checkpointTime: Time)
4444
val sparkConfPairs = ssc.conf.getAll
4545

4646
def createSparkConf(): SparkConf = {
47+
48+
// Reload properties for the checkpoint application since user wants to set a reload property
49+
// or spark had changed its value and user wants to set it back.
50+
val propertiesToReload = List(
51+
"spark.master",
52+
"spark.yarn.keytab",
53+
"spark.yarn.principal")
54+
4755
val newSparkConf = new SparkConf(loadDefaults = false).setAll(sparkConfPairs)
4856
.remove("spark.driver.host")
4957
.remove("spark.driver.port")
50-
val newMasterOption = new SparkConf(loadDefaults = true).getOption("spark.master")
51-
newMasterOption.foreach { newMaster => newSparkConf.setMaster(newMaster) }
58+
val newReloadConf = new SparkConf(loadDefaults = true)
59+
propertiesToReload.foreach { prop =>
60+
newReloadConf.getOption(prop).foreach { value =>
61+
newSparkConf.set(prop, value)
62+
}
63+
}
5264
newSparkConf
5365
}
5466

0 commit comments

Comments
 (0)