HADOOP-19027. S3A: S3AInputStream doesn't recover from channel exceptions #8003

steveloughran · 2025-09-29T17:24:25Z

This is a fraction of #6425 backported to the V1 SDK, which does a lot of V2-specific translation/unwinding.

416 responses are mapped to RangeNotSatisfiableEOFException, whose retry policy is: fail.

Classic EOFException is now retried as connection failure. calls to read() and lazyseek all retry on this with full retry policy, including handling of socket errors.

Everything related to new sdk, its complex http channel stuff: omitted.
This means that the following will be handled

EOF on socket read
connection reset on socket read

Out of range GET is mappped to RangeNotSatisfiableEOFException so it is not retried,
this is critical. Without it ITestS3AOpenCost.testReadPastEOF() will time out as it is continually retrying.

How was this patch tested?

Cherrypick of new ITestS3AOpenCost.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

…xceptions This is a fraction of apache#6425 backported to the V1 SDK, which does a lot of V2-specific translation/unwinding. 416 responses are mapped to RangeNotSatisfiableEOFException, whose retry policy is: fail. Classic EOFException is now retried as connection failure. calls to read() and lazyseek all retry on this with full retry policy, including handling of socket errors.

steveloughran · 2025-09-29T17:43:51Z

S3 London. All good with updated tests.

Failure in ITestAssumeRole which seems unrelated (will need to test on base branch-3.3)
{code}

[ERROR] ITestAssumeRole.testAssumeRoleBadInnerAuth:263->expectFileSystemCreateFailure:137 Expected to find 'not a valid key=value pair (missing equal-sign) in Authorization header' but got unexpected exception: org.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: Invalid key=value pair (missing equal-sign) in Authorization header (hashed with SHA-256 and encoded with Base64): 'hvHJSzfhtkBgtrEOkT9w/UT3So5LWUnDLLsTKYln5vE='. (Service: AWSSecurityTokenService; Status Code: 400; Error Code: IncompleteSignature; Request ID: 4f76f95a-3933-4312-858b-3f1d123bf14f; Proxy: null):IncompleteSignature: Invalid key=value pair (missing equal-sign) in Authorization header (hashed with SHA-256 and encoded with Base64): 'hvHJSzfhtkBgtrEOkT9w/UT3So5LWUnDLLsTKYln5vE='. (Service: AWSSecurityTokenService; Status Code: 400; Error Code: IncompleteSignature; Request ID: 4f76f95a-3933-4312-858b-3f1d123bf14f; Proxy: null)
{code}

hadoop-yetus · 2025-09-29T19:08:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	4m 0s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	35m 20s		branch-3.3 passed
+1 💚	compile	0m 25s		branch-3.3 passed
+1 💚	checkstyle	0m 21s		branch-3.3 passed
+1 💚	mvnsite	0m 30s		branch-3.3 passed
+1 💚	javadoc	0m 30s		branch-3.3 passed
+1 💚	spotbugs	0m 57s		branch-3.3 passed
+1 💚	shadedclient	19m 48s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 24s		the patch passed
+1 💚	compile	0m 19s		the patch passed
-1 ❌	javac	0m 19s	/results-compile-javac-hadoop-tools_hadoop-aws.txt	hadoop-tools_hadoop-aws generated 8 new + 32 unchanged - 0 fixed = 40 total (was 32)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 11s		the patch passed
+1 💚	mvnsite	0m 22s		the patch passed
+1 💚	javadoc	0m 16s		the patch passed
+1 💚	spotbugs	0m 42s		the patch passed
+1 💚	shadedclient	19m 27s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 40s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 28s		The patch does not generate ASF License warnings.
		86m 22s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8003/2/artifact/out/Dockerfile
GITHUB PR	#8003
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux e5a4a63c0131 5.15.0-153-generic #163-Ubuntu SMP Thu Aug 7 16:37:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `a2bd162`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8003/2/testReport/
Max. process+thread count	560 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8003/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-09-29T19:40:12Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	6m 18s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	48m 10s		branch-3.3 passed
+1 💚	compile	0m 39s		branch-3.3 passed
+1 💚	checkstyle	0m 33s		branch-3.3 passed
+1 💚	mvnsite	0m 48s		branch-3.3 passed
+1 💚	javadoc	0m 39s		branch-3.3 passed
+1 💚	spotbugs	1m 11s		branch-3.3 passed
+1 💚	shadedclient	34m 17s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 37s		the patch passed
+1 💚	compile	0m 30s		the patch passed
-1 ❌	javac	0m 30s	/results-compile-javac-hadoop-tools_hadoop-aws.txt	hadoop-tools_hadoop-aws generated 8 new + 32 unchanged - 0 fixed = 40 total (was 32)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 20s		the patch passed
+1 💚	mvnsite	0m 34s		the patch passed
+1 💚	javadoc	0m 25s		the patch passed
+1 💚	spotbugs	1m 7s		the patch passed
+1 💚	shadedclient	33m 35s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	2m 22s	/patch-unit-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch passed.
+1 💚	asflicense	0m 41s		The patch does not generate ASF License warnings.
		134m 2s

Reason	Tests
Failed junit tests	hadoop.fs.s3a.TestS3AExceptionTranslation

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8003/1/artifact/out/Dockerfile
GITHUB PR	#8003
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux e65acc7ece38 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `3b82019`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8003/1/testReport/
Max. process+thread count	732 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8003/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2025-10-03T10:43:49Z

checkstyles are because I added the new SC_ constants, (which deprecates some old ones) but didn't change the uses of those old ones. Which I don't intend to do.

test failure is legit failure of rg.apache.hadoop.fs.s3a.TestS3AExceptionTranslation.test416isEOF

hadoop-yetus · 2025-10-03T16:35:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	6m 32s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ branch-3.3 Compile Tests _
-1 ❌	mvninstall	48m 53s	/branch-mvninstall-root.txt	root in branch-3.3 failed.
+1 💚	compile	0m 39s		branch-3.3 passed
+1 💚	checkstyle	0m 34s		branch-3.3 passed
+1 💚	mvnsite	0m 46s		branch-3.3 passed
+1 💚	javadoc	0m 38s		branch-3.3 passed
+1 💚	spotbugs	1m 17s		branch-3.3 passed
+1 💚	shadedclient	33m 51s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 39s		the patch passed
+1 💚	compile	0m 30s		the patch passed
+1 💚	javac	0m 30s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 20s		the patch passed
+1 💚	mvnsite	0m 36s		the patch passed
+1 💚	javadoc	0m 25s		the patch passed
+1 💚	spotbugs	1m 8s		the patch passed
+1 💚	shadedclient	33m 7s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 20s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		133m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.51 ServerAPI=1.51 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8003/3/artifact/out/Dockerfile
GITHUB PR	#8003
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 29479e31144c 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `b8671f2`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8003/3/testReport/
Max. process+thread count	554 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8003/3/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran added 2 commits September 29, 2025 18:19

HADOOP-19027. Update TestS3AExceptionTranslation for the 416 change

a2bd162

steveloughran changed the title ~~HADOOP-19027. S3A: S3AInputStream doesn't recover from HTTP/channel exceptions~~ HADOOP-19027. S3A: S3AInputStream doesn't recover from channel exceptions Sep 29, 2025

HADOOP-19027. Cut the @Deprecation tags from the existing constants

b8671f2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-19027. S3A: S3AInputStream doesn't recover from channel exceptions #8003

HADOOP-19027. S3A: S3AInputStream doesn't recover from channel exceptions #8003

steveloughran commented Sep 29, 2025

Uh oh!

steveloughran commented Sep 29, 2025

Uh oh!

hadoop-yetus commented Sep 29, 2025

Uh oh!

hadoop-yetus commented Sep 29, 2025

Uh oh!

steveloughran commented Oct 3, 2025

Uh oh!

hadoop-yetus commented Oct 3, 2025

Uh oh!

Uh oh!

HADOOP-19027. S3A: S3AInputStream doesn't recover from channel exceptions #8003

Are you sure you want to change the base?

HADOOP-19027. S3A: S3AInputStream doesn't recover from channel exceptions #8003

Conversation

steveloughran commented Sep 29, 2025

How was this patch tested?

For code changes:

Uh oh!

steveloughran commented Sep 29, 2025

Uh oh!

hadoop-yetus commented Sep 29, 2025

Uh oh!

hadoop-yetus commented Sep 29, 2025

Uh oh!

steveloughran commented Oct 3, 2025

Uh oh!

hadoop-yetus commented Oct 3, 2025

Uh oh!

Uh oh!