[SPARK-5342][YARN] Allow long running Spark apps to run on secure YARN/HDFS #5823

harishreedharan · 2015-05-01T00:35:16Z

Take 2. Does the same thing as #4688, but fixes Hadoop-1 build.

…N/HDFS. Current Spark apps running on Secure YARN/HDFS would not be able to write data to HDFS after 7 days, since delegation tokens cannot be renewed beyond that. This means Spark Streaming apps will not be able to run on Secure YARN. This commit adds basic functionality to fix this issue. In this patch: - new parameters are added - principal and keytab, which can be used to login to a KDC - the client logs in, and then get tokens to start the AM - the keytab is copied to the staging directory - the AM waits for 60% of the time till expiry of the tokens and then logs in using the keytab - each time after 60% of the time, new tokens are created and sent to the executors

…mpl in YarnSparkHadoopUtil.

…n tests.

…ackend is started. Also schedule re-logins in CoarseGrainedSchedulerBackend#start()

…okens()

…nding them over the wire.

… known file's modification time to read the credentials.

…up patch which adds a cleanup mechanism for old credentials files. The credentials files are small and few enough for it to cause issues on HDFS.

…nor fixes.

Conflicts: bin/utils.sh core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala

…les on HDFS.

…kc up the latest suffix from HDFS if the AM is restarted.

…entials files.

…o send tokens via akka on executor startup.

Conflicts: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala

harishreedharan · 2015-05-01T03:55:19Z

Diff from #4688 : harishreedharan/spark@36eb8a9...kerberos-longrunning

SparkQA · 2015-05-01T05:27:53Z

Test build #31507 has finished for PR 5823 at commit b5e7a72.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-05-01T05:34:52Z

Test build #31510 has finished for PR 5823 at commit 4d04301.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

harishreedharan · 2015-05-01T09:26:59Z

Verified that executor and AM failures before and after expiry still work fine.

tgravescs · 2015-05-01T13:54:07Z

looks like the addCredentials method was added in hadoop 2.0.2-alpha so we should be good to use it anywhere in yarn build since we only support 2.2 and on.

tgravescs · 2015-05-01T14:08:53Z

yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala

can you specify each one we are using instead of _

tgravescs · 2015-05-01T14:18:27Z

very minor comments, not even really worth holding this up. I'm going to run some tests and builds manually and if everything looks good commit it.

tgravescs · 2015-05-01T14:30:15Z

seeing the following warning while compiling:

[warn] /home/tgraves/tgravescs_spark/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:247: postfix operator hours should be enabled
[warn] by making the implicit value scala.language.postfixOps visible.
[warn] This can be achieved by adding the import clause 'import scala.language.postfixOps'
[warn] or by setting the compiler option -language:postfixOps.
[warn] See the Scala docs for value scala.language.postfixOps for a discussion
[warn] why the feature should be explicitly enabled.
[warn] sparkConf.getLong("spark.yarn.token.renewal.interval", (24 hours).toMillis)

harishreedharan · 2015-05-01T16:14:20Z

That is the post fix operator warning. I have seen it used elsewhere in
spark but I can switch it out if you want. It trill is not a blocker I
think.

On Friday, May 1, 2015, Tom Graves [email protected] wrote:

seeing the following warning while compiling:

[warn]
/home/tgraves/tgravescs_spark/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:247:
postfix operator hours should be enabled
[warn] by making the implicit value scala.language.postfixOps visible.
[warn] This can be achieved by adding the import clause 'import
scala.language.postfixOps'
[warn] or by setting the compiler option -language:postfixOps.
[warn] See the Scala docs for value scala.language.postfixOps for a
discussion
[warn] why the feature should be explicitly enabled.
[warn] sparkConf.getLong("spark.yarn.token.renewal.interval", (24
hours).toMillis)

—
Reply to this email directly or view it on GitHub
#5823 (comment).

Thanks,
Hari

tgravescs · 2015-05-01T17:18:04Z

its used elsewhere but I think everywhere its used it does:import scala.language.postfixOps

Can you simply add that to the imports so we get rid of the compiler warning.

harishreedharan · 2015-05-01T17:19:05Z

Fixing this and the other imports

Tom Graves wrote:

import scala.language.postfixOps

tgravescs · 2015-05-01T19:23:00Z

+1 pending screwdriver

vanzin · 2015-05-01T19:26:24Z

core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala

super nit: out of order

SparkQA · 2015-05-01T19:26:33Z

Test build #31577 has finished for PR 5823 at commit 3c86bba.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2015-05-01T19:28:58Z

yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala

super nit: in wrong group

vanzin · 2015-05-01T19:40:34Z

Just gave the code a cursory view since the logic has been reviewed ad nauseum. LGTM.

tgravescs · 2015-05-01T19:50:12Z

@vanzin thanks for taking a look. looks like those comments are nits, the exception would be nice but rather then pushing this out further and waiting for jenkins again, I think I'm going to commit this as is and we can file follow up to fix them up. Let me know asap if you object.

vanzin · 2015-05-01T19:53:32Z

Yeah, they're all nits, with the exception of the exception (which you could easily add when pushing if you feel like it).

tgravescs · 2015-05-01T20:33:09Z

changed to pass exception to logWarning on commit.

…RN/HDFS Take 2. Does the same thing as apache#4688, but fixes Hadoop-1 build. Author: Hari Shreedharan <[email protected]> Closes apache#5823 from harishreedharan/kerberos-longrunning and squashes the following commits: 3c86bba [Hari Shreedharan] Import fixes. Import postfixOps explicitly. 4d04301 [Hari Shreedharan] Minor formatting fixes. b5e7a72 [Hari Shreedharan] Remove reflection, use a method in SparkHadoopUtil to update the token renewer. 7bff6e9 [Hari Shreedharan] Make sure all required classes are present in the jar. Fix import order. e851f70 [Hari Shreedharan] Move the ExecutorDelegationTokenRenewer to yarn module. Use reflection to use it. 36eb8a9 [Hari Shreedharan] Change the renewal interval config param. Fix a bunch of comments. 611923a [Hari Shreedharan] Make sure the namenodes are listed correctly for creating tokens. 09fe224 [Hari Shreedharan] Use token.renew to get token's renewal interval rather than using hdfs-site.xml 6963bbc [Hari Shreedharan] Schedule renewal in AM before starting user class. Else, a restarted AM cannot access HDFS if the user class tries to. 072659e [Hari Shreedharan] Fix build failure caused by thread factory getting moved to ThreadUtils. f041dd3 [Hari Shreedharan] Merge branch 'master' into kerberos-longrunning 42eead4 [Hari Shreedharan] Remove RPC part. Refactor and move methods around, use renewal interval rather than max lifetime to create new tokens. ebb36f5 [Hari Shreedharan] Merge branch 'master' into kerberos-longrunning bc083e3 [Hari Shreedharan] Overload RegisteredExecutor to send tokens. Minor doc updates. 7b19643 [Hari Shreedharan] Merge branch 'master' into kerberos-longrunning 8a4f268 [Hari Shreedharan] Added docs in the security guide. Changed some code to ensure that the renewer objects are created only if required. e800c8b [Hari Shreedharan] Restore original RegisteredExecutor message, and send new tokens via NewTokens message. 0e9507e [Hari Shreedharan] Merge branch 'master' into kerberos-longrunning 7f1bc58 [Hari Shreedharan] Minor fixes, cleanup. bcd11f9 [Hari Shreedharan] Refactor AM and Executor token update code into separate classes, also send tokens via akka on executor startup. f74303c [Hari Shreedharan] Move the new logic into specialized classes. Add cleanup for old credentials files. 2f9975c [Hari Shreedharan] Ensure new tokens are written out immediately on AM restart. Also, pikc up the latest suffix from HDFS if the AM is restarted. 61b2b27 [Hari Shreedharan] Account for AM restarts by making sure lastSuffix is read from the files on HDFS. 62c45ce [Hari Shreedharan] Relogin from keytab periodically. fa233bd [Hari Shreedharan] Adding logging, fixing minor formatting and ordering issues. 42813b4 [Hari Shreedharan] Remove utils.sh, which was re-added due to merge with master. 0de27ee [Hari Shreedharan] Merge branch 'master' into kerberos-longrunning 55522e3 [Hari Shreedharan] Fix failure caused by Preconditions ambiguity. 9ef5f1b [Hari Shreedharan] Added explanation of how the credentials refresh works, some other minor fixes. f4fd711 [Hari Shreedharan] Fix SparkConf usage. 2debcea [Hari Shreedharan] Change the file structure for credentials files. I will push a followup patch which adds a cleanup mechanism for old credentials files. The credentials files are small and few enough for it to cause issues on HDFS. af6d5f0 [Hari Shreedharan] Cleaning up files where changes weren't required. f0f54cb [Hari Shreedharan] Be more defensive when updating the credentials file. f6954da [Hari Shreedharan] Got rid of Akka communication to renew, instead the executors check a known file's modification time to read the credentials. 5c11c3e [Hari Shreedharan] Move tests to YarnSparkHadoopUtil to fix compile issues. b4cb917 [Hari Shreedharan] Send keytab to AM via DistributedCache rather than directly via HDFS 0985b4e [Hari Shreedharan] Write tokens to HDFS and read them back when required, rather than sending them over the wire. d79b2b9 [Hari Shreedharan] Make sure correct credentials are passed to FileSystem#addDelegationTokens() 8c6928a [Hari Shreedharan] Fix issue caused by direct creation of Actor object. fb27f46 [Hari Shreedharan] Make sure principal and keytab are set before CoarseGrainedSchedulerBackend is started. Also schedule re-logins in CoarseGrainedSchedulerBackend#start() 41efde0 [Hari Shreedharan] Merge branch 'master' into kerberos-longrunning d282d7a [Hari Shreedharan] Fix ClientSuite to set YARN mode, so that the correct class is used in tests. bcfc374 [Hari Shreedharan] Fix Hadoop-1 build by adding no-op methods in SparkHadoopUtil, with impl in YarnSparkHadoopUtil. f8fe694 [Hari Shreedharan] Handle None if keytab-login is not scheduled. 2b0d745 [Hari Shreedharan] [SPARK-5342][YARN] Allow long running Spark apps to run on secure YARN/HDFS. ccba5bc [Hari Shreedharan] WIP: More changes wrt kerberos 77914dd [Hari Shreedharan] WIP: Add kerberos principal and keytab to YARN client.

harishreedharan added 30 commits February 18, 2015 22:00

WIP: Add kerberos principal and keytab to YARN client.

77914dd

WIP: More changes wrt kerberos

ccba5bc

Handle None if keytab-login is not scheduled.

f8fe694

Fix Hadoop-1 build by adding no-op methods in SparkHadoopUtil, with i…

bcfc374

…mpl in YarnSparkHadoopUtil.

Fix ClientSuite to set YARN mode, so that the correct class is used i…

d282d7a

…n tests.

Merge branch 'master' into kerberos-longrunning

41efde0

Make sure principal and keytab are set before CoarseGrainedSchedulerB…

fb27f46

…ackend is started. Also schedule re-logins in CoarseGrainedSchedulerBackend#start()

Fix issue caused by direct creation of Actor object.

8c6928a

Make sure correct credentials are passed to FileSystem#addDelegationT…

d79b2b9

…okens()

Write tokens to HDFS and read them back when required, rather than se…

0985b4e

…nding them over the wire.

Send keytab to AM via DistributedCache rather than directly via HDFS

b4cb917

Move tests to YarnSparkHadoopUtil to fix compile issues.

5c11c3e

Got rid of Akka communication to renew, instead the executors check a…

f6954da

… known file's modification time to read the credentials.

Be more defensive when updating the credentials file.

f0f54cb

Cleaning up files where changes weren't required.

af6d5f0

Change the file structure for credentials files. I will push a follow…

2debcea

…up patch which adds a cleanup mechanism for old credentials files. The credentials files are small and few enough for it to cause issues on HDFS.

Fix SparkConf usage.

f4fd711

Added explanation of how the credentials refresh works, some other mi…

9ef5f1b

…nor fixes.

Fix failure caused by Preconditions ambiguity.

55522e3

Merge branch 'master' into kerberos-longrunning

0de27ee

Conflicts: bin/utils.sh core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala

Remove utils.sh, which was re-added due to merge with master.

42813b4

Adding logging, fixing minor formatting and ordering issues.

fa233bd

Relogin from keytab periodically.

62c45ce

Account for AM restarts by making sure lastSuffix is read from the fi…

61b2b27

…les on HDFS.

Ensure new tokens are written out immediately on AM restart. Also, pi…

2f9975c

…kc up the latest suffix from HDFS if the AM is restarted.

Move the new logic into specialized classes. Add cleanup for old cred…

f74303c

…entials files.

Refactor AM and Executor token update code into separate classes, als…

bcd11f9

…o send tokens via akka on executor startup.

Minor fixes, cleanup.

7f1bc58

tgravescs reviewed May 1, 2015
View reviewed changes

Import fixes. Import postfixOps explicitly.

3c86bba

vanzin reviewed May 1, 2015
View reviewed changes

core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala

Copy link

Contributor

vanzin May 1, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

super nit: out of order

vanzin reviewed May 1, 2015
View reviewed changes

yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala

Copy link

Contributor

vanzin May 1, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

super nit: in wrong group

asfgit closed this in b1f4ca8 May 1, 2015

[SPARK-5342][YARN] Allow long running Spark apps to run on secure YARN/HDFS #5823

[SPARK-5342][YARN] Allow long running Spark apps to run on secure YARN/HDFS #5823

Uh oh!

Conversation

harishreedharan commented May 1, 2015

Uh oh!

harishreedharan commented May 1, 2015

Uh oh!

SparkQA commented May 1, 2015

Uh oh!

SparkQA commented May 1, 2015

Uh oh!

harishreedharan commented May 1, 2015

Uh oh!

tgravescs commented May 1, 2015

Uh oh!

tgravescs May 1, 2015

Choose a reason for hiding this comment

Uh oh!

tgravescs commented May 1, 2015

Uh oh!

tgravescs commented May 1, 2015

Uh oh!

harishreedharan commented May 1, 2015

Uh oh!

tgravescs commented May 1, 2015

Uh oh!

harishreedharan commented May 1, 2015

Uh oh!

tgravescs commented May 1, 2015

Uh oh!

vanzin May 1, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 1, 2015

Uh oh!

vanzin May 1, 2015

Choose a reason for hiding this comment

Uh oh!

vanzin commented May 1, 2015

Uh oh!

tgravescs commented May 1, 2015

Uh oh!

vanzin commented May 1, 2015

Uh oh!

tgravescs commented May 1, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants