HADOOP-19535: S3A: Support WebIdentityTokenFileCredentialsProvider #7802

shameersss1 · 2025-07-14T06:25:45Z

Description of PR

Support For Adding S3 Credential providers like WebIdentityTokenFileCredentialsProvider which can throw other than SDKException when failing to resolve credentials

How was this patch tested?

Tested in us-east-1 by running Unit tests and Integration tests mvn -Dparallel-tests -DtestsThreadCount=8 clean verify
Tested in Amazon EC2 , Amazon EKS / K8s , Amazon Container / EMR Serverless

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

…fault S3 credential provider chain

steveloughran

-1 to adding to the default chain, it changes behaviour in a way which isn't obvious to anyone looking at a system.

if you deploy on an older release, auth will fail. asking for an explicit declaration of the provider makes it work everywhere.

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/AWSCredentialProviderList.java

steveloughran · 2025-07-14T18:37:48Z

...ls/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/CredentialProviderListFactory.java

        EC2_IAM_CREDENTIALS_V2);
    v1v2CredentialProviderMap.put(ENVIRONMENT_CREDENTIALS_V1,
        ENVIRONMENT_CREDENTIALS_V2);
+    v1v2CredentialProviderMap.put(WEB_IDENTITY_CREDENTIALS_V1,


no need to care about v1 to v2 migration; this wasn't something in use. if it was, we'd have had complaints about

shameersss1 · 2025-07-15T02:15:17Z

@steveloughran - I have omitted the default credential provider chain changes and kept the rest to make the current AWSCredentialProviderList.java compatible with WebIdentityTokenFileCredentialsProvider or anyother credentials provider which can throw exception other than SDKException

steveloughran

ok, but see comment about logging

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/AWSCredentialProviderList.java

hadoop-yetus · 2025-07-16T01:11:46Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	20m 29s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	44m 12s		Maven dependency ordering for branch
-1 ❌	mvninstall	33m 4s	/branch-mvninstall-root.txt	root in trunk failed.
+1 💚	compile	17m 37s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	15m 40s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	4m 44s		trunk passed
+1 💚	mvnsite	2m 37s		trunk passed
+1 💚	javadoc	2m 6s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 35s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	4m 3s		trunk passed
+1 💚	shadedclient	39m 20s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 22s		the patch passed
+1 💚	compile	14m 57s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	14m 57s		the patch passed
+1 💚	compile	14m 38s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	14m 38s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 37s	/results-checkstyle-root.txt	root: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	mvnsite	2m 34s		the patch passed
+1 💚	javadoc	1m 56s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 37s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	4m 48s		the patch passed
+1 💚	shadedclient	43m 17s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	23m 28s		hadoop-common in the patch passed.
+1 💚	unit	3m 36s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 59s		The patch does not generate ASF License warnings.
		307m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7802/1/artifact/out/Dockerfile
GITHUB PR	#7802
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle markdownlint
uname	Linux b64948dea634 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `dbb0e4f`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7802/1/testReport/
Max. process+thread count	1266 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7802/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-07-16T10:24:28Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 59s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	60m 34s		Maven dependency ordering for branch
+1 💚	mvninstall	32m 50s		trunk passed
+1 💚	compile	15m 56s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	14m 3s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	4m 9s		trunk passed
+1 💚	mvnsite	2m 43s		trunk passed
+1 💚	javadoc	2m 19s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 48s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	3m 57s		trunk passed
+1 💚	shadedclient	36m 30s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 34s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 29s		the patch passed
+1 💚	compile	15m 7s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	15m 7s		the patch passed
+1 💚	compile	14m 13s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	14m 13s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 10s		the patch passed
+1 💚	mvnsite	2m 40s		the patch passed
+1 💚	javadoc	2m 6s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 47s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	4m 23s		the patch passed
+1 💚	shadedclient	36m 41s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	23m 9s		hadoop-common in the patch passed.
+1 💚	unit	3m 48s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 8s		The patch does not generate ASF License warnings.
		291m 29s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7802/2/artifact/out/Dockerfile
GITHUB PR	#7802
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle markdownlint
uname	Linux adabaf932674 5.15.0-143-generic #153-Ubuntu SMP Fri Jun 13 19:10:45 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `1ed0984`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7802/2/testReport/
Max. process+thread count	1266 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7802/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

shameersss1 · 2025-07-21T06:53:17Z

@steveloughran - Are we good to merge this ?

steveloughran

LGTM
+1

steveloughran · 2025-07-21T17:09:52Z

@shameersss1 merged. for a 3.4.x backport, I see we don't have that authentication.md file.

Options

skip that doc change
add it where the branch's auth provider list goes
pull in aull of authentication.md

IMO option #2 is the best combination of ease and utility.

shameersss1 · 2025-07-22T12:31:59Z

+1 for option 2

steveloughran · 2025-07-22T16:56:34Z

ok, do that and I'll merge without any review, though of course I expect a test run...

HADOOP-19535: S3A : Add WebIdentityTokenFileCredentialsProvider to de…

4725fb1

…fault S3 credential provider chain

github-actions bot added trunk Common TOOLS AWS labels Jul 14, 2025

steveloughran reviewed Jul 14, 2025

View reviewed changes

Adressing Review comments

dbb0e4f

shameersss1 changed the title ~~HADOOP-19535: S3A : Add WebIdentityTokenFileCredentialsProvider to default S3 credential provider chain~~ HADOOP-19535: S3A : Support For Adding WebIdentityTokenFileCredentialsProvider to default S3 credential provider chain Jul 15, 2025

shameersss1 requested a review from steveloughran July 15, 2025 02:15

steveloughran requested changes Jul 15, 2025

View reviewed changes

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/AWSCredentialProviderList.java Show resolved Hide resolved

Addressing PR comments and checkstyle issue

1ed0984

shameersss1 requested a review from steveloughran July 16, 2025 05:31

steveloughran approved these changes Jul 21, 2025

View reviewed changes

steveloughran merged commit ca25b1b into apache:trunk Jul 21, 2025
4 checks passed

steveloughran changed the title ~~HADOOP-19535: S3A : Support For Adding WebIdentityTokenFileCredentialsProvider to default S3 credential provider chain~~ HADOOP-19535: S3A: Support WebIdentityTokenFileCredentialsProvider Jul 21, 2025

HADOOP-19535: S3A: Support WebIdentityTokenFileCredentialsProvider #7802

HADOOP-19535: S3A: Support WebIdentityTokenFileCredentialsProvider #7802

Uh oh!

Conversation

shameersss1 commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

steveloughran Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

shameersss1 Jul 15, 2025

Choose a reason for hiding this comment

Uh oh!

shameersss1 commented Jul 15, 2025

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hadoop-yetus commented Jul 16, 2025

Uh oh!

hadoop-yetus commented Jul 16, 2025

Uh oh!

shameersss1 commented Jul 21, 2025

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

steveloughran commented Jul 21, 2025

Uh oh!

shameersss1 commented Jul 22, 2025

Uh oh!

steveloughran commented Jul 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shameersss1 commented Jul 14, 2025 •

edited

Loading