Backport Hadoop 17215 branch 2.10 #4261

arjun4084346 · 2022-05-03T16:17:45Z

Description of PR

The reason for this cherry pick is to backport fix for https://issues.apache.org/jira/browse/HADOOP-17215 .The following commits were cherry picked in order to do it cleanly

8031c66
459eb2a
7f486f0
3472c3e
0d85515
e31a636
07b7d07

The second last commit e31a636 was the intended one and the last one was cherry picked because it fixed some test failures.

How was this patch tested?

Ran mvn test -pl hadoop-tools/hadoop-azure

[INFO] Results:
[INFO] 
[WARNING] Tests run: 277, Failures: 0, Errors: 0, Skipped: 4
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  06:09 min
[INFO] Finished at: 2022-05-03T09:43:05-07:00
[INFO] ------------------------------------------------------------------------

Ran all integration abfs tests using mvn -T 1C -Dparallel-tests=abfs clean verify with my storage account arjundev.dfs.core.windows.net
Storage account's Primary location: East US, Secondary location: West US

[INFO] Results:
[INFO] 
[INFO] Tests run: 42, Failures: 0, Errors: 0, Skipped: 0

[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   ITestAzureBlobFileSystemCreate.testFilterFSWriteAfterClose:210 » IO java.io.Fi...
[ERROR]   ITestAzureBlobFileSystemCreate.testNegativeScenariosForCreateOverwriteDisabled:363 » TokenAccessProvider
[ERROR]   ITestAzureBlobFileSystemE2E.testFlushWithFileNotFoundException:224 » IO java.i...
[ERROR]   ITestAzureBlobFileSystemE2E.testWriteWithFileNotFoundException:204 » IO java.i...
[INFO] 
[ERROR] Tests run: 426, Failures: 0, Errors: 4, Skipped: 253

[INFO] Results:
[INFO] 
[WARNING] Tests run: 151, Failures: 0, Errors: 0, Skipped: 24

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

Contributed by Mukund Thakur. (cherry picked from commit 8031c66)

Contributed by Mehakmeet Singh.There (cherry picked from commit 459eb2a)

Contributed by: Mehakmeet Singh. Change-Id: Ib84e7a42f28e064df4c6204fcce33e573360bf42 (cherry picked from commit 7f486f0)

Contributed by Mehakmeet Singh. (cherry picked from commit 3472c3e)

…nse failure occurs in ABFS (apache#2264) Contributed by Mehakmeet Singh (cherry picked from commit 0d85515)

Contributed by Sneha Vijayarajan DETAILS: This change adds config key "fs.azure.enable.conditional.create.overwrite" with a default of true. When enabled, if create(path, overwrite: true) is invoked and the file exists, the ABFS driver will first obtain its etag and then attempt to overwrite the file on the condition that the etag matches. The purpose of this is to mitigate the non-idempotency of this method. Specifically, in the event of a network error or similar, the client will retry and this can result in the file being created more than once which may result in data loss. In essense this is like a poor man's file handle, and will be addressed more thoroughly in the future when support for lease is added to ABFS. TEST RESULTS: namespace.enabled=true auth.type=SharedKey ------------------- $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 42 Tests run: 207, Failures: 0, Errors: 0, Skipped: 24 namespace.enabled=true auth.type=OAuth ------------------- $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 74 Tests run: 207, Failures: 0, Errors: 0, Skipped: 140 (cherry picked from commit e31a636)

Contributed by Ayush Saxena and Steve Loughran Change-Id: I4bb76815bc1d11d1804dc67bafde68b6a995b974 (cherry picked from commit 07b7d07)

hadoop-yetus · 2022-05-03T18:33:59Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	11m 54s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 11 new or modified test files.
			_ branch-2.10 Compile Tests _
+0 🆗	mvndep	3m 51s		Maven dependency ordering for branch
+1 💚	mvninstall	14m 23s		branch-2.10 passed
+1 💚	compile	14m 5s		branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	compile	11m 34s		branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚	checkstyle	2m 25s		branch-2.10 passed
+1 💚	mvnsite	2m 31s		branch-2.10 passed
+1 💚	javadoc	2m 25s		branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javadoc	2m 2s		branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
-1 ❌	spotbugs	2m 42s	/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html	hadoop-common-project/hadoop-common in branch-2.10 has 2 extant spotbugs warnings.
-0 ⚠️	patch	1m 37s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 20s		the patch passed
+1 💚	compile	13m 18s		the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javac	13m 18s		the patch passed
+1 💚	compile	11m 39s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚	javac	11m 39s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	2m 21s	/results-checkstyle-root.txt	root: The patch generated 6 new + 8 unchanged - 0 fixed = 14 total (was 8)
+1 💚	mvnsite	2m 27s		the patch passed
-1 ❌	javadoc	0m 50s	/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkAzulSystems,Inc.-1.7.0_262-b10.txt	hadoop-tools_hadoop-azure-jdkAzulSystems,Inc.-1.7.0_262-b10 with JDK Azul Systems, Inc.-1.7.0_262-b10 generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3)
+1 💚	javadoc	2m 2s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚	spotbugs	3m 55s		the patch passed
			_ Other Tests _
+1 💚	unit	16m 58s		hadoop-common in the patch passed.
-1 ❌	unit	1m 46s	/patch-unit-hadoop-tools_hadoop-azure.txt	hadoop-azure in the patch passed.
+1 💚	asflicense	1m 5s		The patch does not generate ASF License warnings.
		135m 20s

Reason	Tests
Failed junit tests	hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
	hadoop.fs.azure.TestNativeAzureFileSystemMocked
	hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
	hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
	hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
	hadoop.fs.azure.TestOutOfBandAzureBlobOperations
	hadoop.fs.azure.TestWasbFsck
	hadoop.fs.azure.TestBlobMetadata

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4261/1/artifact/out/Dockerfile
GITHUB PR	#4261
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 9cbed676f5a6 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-2.10 / `7843368`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Multi-JDK versions	/usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4261/1/testReport/
Max. process+thread count	1349 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4261/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-05-03T19:16:14Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 14 new or modified test files.
			_ branch-2.10 Compile Tests _
+0 🆗	mvndep	4m 3s		Maven dependency ordering for branch
+1 💚	mvninstall	14m 0s		branch-2.10 passed
+1 💚	compile	13m 30s		branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	compile	11m 23s		branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚	checkstyle	2m 31s		branch-2.10 passed
+1 💚	mvnsite	2m 58s		branch-2.10 passed
+1 💚	javadoc	2m 45s		branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javadoc	2m 33s		branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
-1 ❌	spotbugs	2m 50s	/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html	hadoop-common-project/hadoop-common in branch-2.10 has 2 extant spotbugs warnings.
-0 ⚠️	patch	1m 55s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 26s		the patch passed
+1 💚	compile	12m 33s		the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javac	12m 33s		the patch passed
+1 💚	compile	11m 28s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚	javac	11m 28s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	2m 25s	/results-checkstyle-root.txt	root: The patch generated 6 new + 25 unchanged - 0 fixed = 31 total (was 25)
+1 💚	mvnsite	2m 54s		the patch passed
-1 ❌	javadoc	1m 3s	/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkAzulSystems,Inc.-1.7.0_262-b10.txt	hadoop-tools_hadoop-azure-jdkAzulSystems,Inc.-1.7.0_262-b10 with JDK Azul Systems, Inc.-1.7.0_262-b10 generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3)
+1 💚	javadoc	2m 30s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚	spotbugs	4m 16s		the patch passed
			_ Other Tests _
-1 ❌	unit	9m 58s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch passed.
+1 💚	unit	2m 1s		hadoop-azure in the patch passed.
+1 💚	asflicense	1m 17s		The patch does not generate ASF License warnings.
		132m 56s

Reason	Tests
Failed junit tests	hadoop.util.TestReadWriteDiskValidator

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4261/2/artifact/out/Dockerfile
GITHUB PR	#4261
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux c575f4115eec 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-2.10 / `2280569`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Multi-JDK versions	/usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4261/2/testReport/
Max. process+thread count	1367 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4261/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

arjun4084346 · 2022-05-11T21:19:13Z

@steveloughran please review/merge this. Created this one PR to cherry pick seven commits as discussed in #4171

github-actions · 2025-11-12T00:23:20Z

We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
Thanks all for your contribution.

mukund-thakur and others added 7 commits May 3, 2022 09:16

HADOOP-16965. Refactor abfs stream configuration. (apache#1956)

b03b211

Contributed by Mukund Thakur. (cherry picked from commit 8031c66)

HADOOP-16914 Adding Output Stream Counters in ABFS (apache#1899)

957e75d

Contributed by Mehakmeet Singh.There (cherry picked from commit 459eb2a)

HADOOP-17016. Adding Common Counters in ABFS (apache#1991).

6fa99fa

Contributed by: Mehakmeet Singh. Change-Id: Ib84e7a42f28e064df4c6204fcce33e573360bf42 (cherry picked from commit 7f486f0)

HADOOP-17065. Add Network Counters to ABFS (apache#2056)

0ca5415

Contributed by Mehakmeet Singh. (cherry picked from commit 3472c3e)

HADOOP-17229. No updation of bytes received counter value after respo…

75a398d

…nse failure occurs in ABFS (apache#2264) Contributed by Mehakmeet Singh (cherry picked from commit 0d85515)

HADOOP-17325. WASB Test Failures

2280569

Contributed by Ayush Saxena and Steve Loughran Change-Id: I4bb76815bc1d11d1804dc67bafde68b6a995b974 (cherry picked from commit 07b7d07)

arjun4084346 mentioned this pull request May 3, 2022

HADOOP-16965. Refactor abfs stream configuration. (#1956) #4171

Closed

4 tasks

arjun4084346 changed the title ~~Hadoop 17215 branch 2.10~~ Backport Hadoop 17215 branch 2.10 May 11, 2022

github-actions bot added the Stale label Nov 12, 2025

github-actions bot closed this Nov 13, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Backport Hadoop 17215 branch 2.10 #4261

Backport Hadoop 17215 branch 2.10 #4261

Uh oh!

arjun4084346 commented May 3, 2022 •

edited

Loading

Uh oh!

hadoop-yetus commented May 3, 2022

Uh oh!

hadoop-yetus commented May 3, 2022

Uh oh!

arjun4084346 commented May 11, 2022

Uh oh!

github-actions bot commented Nov 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Backport Hadoop 17215 branch 2.10 #4261

Backport Hadoop 17215 branch 2.10 #4261

Uh oh!

Conversation

arjun4084346 commented May 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented May 3, 2022

Uh oh!

hadoop-yetus commented May 3, 2022

Uh oh!

arjun4084346 commented May 11, 2022

Uh oh!

github-actions bot commented Nov 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

arjun4084346 commented May 3, 2022 •

edited

Loading