[SPARK-26766][CORE] Remove the list of filesystems from HadoopDelegationTokenProvider.obtainDelegationTokens #23698
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ionTokenProvider.obtainDelegationTokens
|
Test build #101891 has finished for PR 23698 at commit
|
|
Test build #101895 has finished for PR 23698 at commit
|
vanzin
reviewed
Jan 30, 2019
Contributor
vanzin
left a comment
vanzin
left a comment
There was a problem hiding this comment.
You'll need to update the docs that reference the settings you're renaming.
core/src/main/scala/org/apache/spark/deploy/security/HadoopFSDelegationTokenProvider.scala
Outdated
Show resolved
Hide resolved
core/src/main/scala/org/apache/spark/deploy/security/HadoopFSDelegationTokenProvider.scala
Outdated
Show resolved
Hide resolved
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala
Show resolved
Hide resolved
core/src/main/scala/org/apache/spark/deploy/security/HadoopFSDelegationTokenProvider.scala
Show resolved
Hide resolved
* Config parameter deprecation
* Return defaultFS all the time
* get("spark.master", null)
|
Test build #101959 has finished for PR 23698 at commit
|
|
Test build #101960 has finished for PR 23698 at commit
|
vanzin
reviewed
Jan 31, 2019
core/src/main/scala/org/apache/spark/deploy/security/HadoopFSDelegationTokenProvider.scala
Outdated
Show resolved
Hide resolved
core/src/main/scala/org/apache/spark/internal/config/package.scala
Outdated
Show resolved
Hide resolved
Contributor
|
Still missing the doc update. |
* Doc update * Param deprecation
Contributor
Author
Yeah, left from the last commit :/ Now updated. |
|
Test build #101994 has finished for PR 23698 at commit
|
vanzin
reviewed
Feb 6, 2019
* Simplified hadoopFSsToAccess * Moved doc to generic area
|
Test build #102077 has finished for PR 23698 at commit
|
Contributor
|
retest this please |
|
Test build #102082 has finished for PR 23698 at commit
|
Contributor
|
Merging to master. |
jackylee-ch
pushed a commit
to jackylee-ch/spark
that referenced
this pull request
Feb 18, 2019
…ionTokenProvider.obtainDelegationTokens ## What changes were proposed in this pull request? Delegation token providers interface now has a parameter `fileSystems` but this is needed only for `HadoopFSDelegationTokenProvider`. In this PR I've addressed this issue in the following way: * Removed `fileSystems` parameter from `HadoopDelegationTokenProvider` * Moved `YarnSparkHadoopUtil.hadoopFSsToAccess` into `HadoopFSDelegationTokenProvider` * Moved `spark.yarn.stagingDir` into core * Moved `spark.yarn.access.namenodes` into core and renamed to `spark.kerberos.access.namenodes` * Moved `spark.yarn.access.hadoopFileSystems` into core and renamed to `spark.kerberos.access.hadoopFileSystems` ## How was this patch tested? Existing unit tests. Closes apache#23698 from gaborgsomogyi/SPARK-26766. Authored-by: Gabor Somogyi <[email protected]> Signed-off-by: Marcelo Vanzin <[email protected]>
turboFei
pushed a commit
to turboFei/spark
that referenced
this pull request
Nov 6, 2025
…nfigs of Hadoop Filesystems to access (apache#245) [HADP-45851] Fix backward compatibility of alternative configs of Hadoop Filesystems to access (apache#119) ### What changes were proposed in this pull request? Fix precedence of configs of Hadoop Filesystems to access. Before this PR ``` spark.kerberos.access.hadoopFileSystems -> spark.yarn.access.namenodes -> spark.yarn.access.hadoopFileSystems ``` After this PR ``` spark.kerberos.access.hadoopFileSystems -> spark.yarn.access.hadoopFileSystems -> spark.yarn.access.namenodes ``` ### Why are the changes needed? Before apache#23698, the precedence of configuring Hadoop Filesystems to access is ``` spark.yarn.access.hadoopFileSystems -> spark.yarn.access.namenodes ``` Afterwards, it's ``` spark.kerberos.access.hadoopFileSystems -> spark.yarn.access.namenodes -> spark.yarn.access.hadoopFileSystems ``` When both `spark.yarn.access.hadoopFileSystems` and `spark.yarn.access.namenodes` are configured with different values, the PR will break backward compatibility and cause application failure. ### Does this PR introduce _any_ user-facing change? Yes. Fix backward compatibility. ### How was this patch tested? Updated UT. Co-authored-by: tianlzhang <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Delegation token providers interface now has a parameter
fileSystemsbut this is needed only forHadoopFSDelegationTokenProvider.In this PR I've addressed this issue in the following way:
fileSystemsparameter fromHadoopDelegationTokenProviderYarnSparkHadoopUtil.hadoopFSsToAccessintoHadoopFSDelegationTokenProviderspark.yarn.stagingDirinto corespark.yarn.access.namenodesinto core and renamed tospark.kerberos.access.namenodesspark.yarn.access.hadoopFileSystemsinto core and renamed tospark.kerberos.access.hadoopFileSystemsHow was this patch tested?
Existing unit tests.