HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const #2795

sumangala-patki · 2021-03-22T12:04:09Z

The default value of readahead queue depth is currently set to the number of available processors. However, this can result in one inputstream instance consuming more processor time. To ensure equal thread allocation during read for all inputstreams created in a session, we change the default readahead queue depth to a constant (2).

hadoop-yetus · 2021-03-22T12:06:51Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 17s		#2795 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem	Report/Notes
GITHUB PR	#2795
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/1/console
versions	git=2.17.1
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-03-23T06:05:09Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 3s		trunk passed
+1 💚	compile	0m 36s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	0m 33s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	0m 26s		trunk passed
+1 💚	mvnsite	0m 39s		trunk passed
+1 💚	javadoc	0m 31s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 28s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	0m 58s		trunk passed
+1 💚	shadedclient	14m 33s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	14m 51s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 30s		the patch passed
+1 💚	compile	0m 31s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	0m 31s		the patch passed
+1 💚	compile	0m 26s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	0m 26s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 16s		the patch passed
+1 💚	mvnsite	0m 29s		the patch passed
+1 💚	javadoc	0m 23s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 20s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	1m 2s		the patch passed
+1 💚	shadedclient	14m 48s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 7s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 31s		The patch does not generate ASF License warnings.
		75m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/2/artifact/out/Dockerfile
GITHUB PR	#2795
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 75564054a6e1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b165323`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/2/testReport/
Max. process+thread count	665 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

sumangala-patki · 2021-03-23T06:35:31Z

TEST RESULTS

HNS Account Location: East US 2
NonHNS Account Location: East US 2, Central US

HNS OAuth

[INFO] Tests run: 94, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 497, Failures: 0, Errors: 0, Skipped: 70
[WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 49

HNS-SharedKey

[INFO] Tests run: 94, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 495, Failures: 0, Errors: 0, Skipped: 24
[WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 40

Non-HNS SharedKey

[INFO] Tests run: 94, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 504, Failures: 0, Errors: 0, Skipped: 250
[WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 40

hadoop-yetus · 2021-03-26T08:22:50Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	13m 16s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	33m 38s		trunk passed
+1 💚	compile	0m 38s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	0m 37s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	0m 25s		trunk passed
+1 💚	mvnsite	0m 40s		trunk passed
+1 💚	javadoc	0m 33s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 28s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	1m 2s		trunk passed
+1 💚	shadedclient	14m 15s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	14m 34s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s		the patch passed
+1 💚	compile	0m 30s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	0m 30s		the patch passed
+1 💚	compile	0m 25s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	0m 25s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 17s		the patch passed
+1 💚	mvnsite	0m 28s		the patch passed
+1 💚	javadoc	0m 22s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 21s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	0m 59s		the patch passed
+1 💚	shadedclient	13m 44s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 8s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		87m 8s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/3/artifact/out/Dockerfile
GITHUB PR	#2795
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint
uname	Linux 1a2a6de7b3dd 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b8fc345`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/3/testReport/
Max. process+thread count	536 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

snvijaya

+1

...s/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java

anoopsjohn · 2021-05-13T06:55:20Z

When there are not many parallel FIS reads, this (possible) reduction in read ahead parallelism will affect the read throughput? Specially Spark kind of? From HBase standpoint this is acceptable.

hadoop-yetus · 2021-05-13T11:25:46Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 33s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 11s		trunk passed
+1 💚	compile	0m 39s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	0m 36s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	0m 27s		trunk passed
+1 💚	mvnsite	0m 41s		trunk passed
+1 💚	javadoc	0m 32s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 30s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	0m 59s		trunk passed
+1 💚	shadedclient	14m 3s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	14m 21s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s		the patch passed
+1 💚	compile	0m 31s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	0m 31s		the patch passed
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	0m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 16s		the patch passed
+1 💚	mvnsite	0m 29s		the patch passed
+1 💚	javadoc	0m 23s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 21s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	1m 0s		the patch passed
+1 💚	shadedclient	13m 51s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 8s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		75m 0s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/4/artifact/out/Dockerfile
GITHUB PR	#2795
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint
uname	Linux 10f225a42410 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `56635ba`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/4/testReport/
Max. process+thread count	689 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2795/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

vinaysbadami · 2021-05-17T05:33:04Z

@anoopsjohn ==> wrt your comment on "will this hamper spark jobs".

This is a speculative read ahead. So whenever this is wrong, it is waste io and iops.
With parquet etc, the reads tend to be random. Hence a smaller read ahead depth is prefereable
Based on debugging various customer perf isues, we rarely saw benefit of > 2
This config is per inputstream and not global across streams.

anoopsjohn · 2021-05-17T07:03:37Z

Thanks @vinaysbadami . Ya speculative part I agree. Points #2 and #3 are very important. You can think whether can include this in abfs.md even. Will be good.

This config is per inputstream and not global across streams.

No I dont think so. This config is cluster level and all input streams opened in this will use this value.
Now we have openFile(Path path) API usage in AzureBlobFileSystem. This will allow passing per input stream options. But this config support is not added yet.

anoopsjohn · 2021-05-17T07:04:25Z

This will allow passing per input stream options. But this config support is not added yet.

We can follow up in a jira. Will be nice to have as a feature support from Driver side.
+1 for this PR.

surendralilhore

+1

surendralilhore · 2021-06-03T08:47:16Z

Thanks @anoopsjohn for review

steveloughran · 2021-06-08T20:30:54Z

Now this has been merged, the JIRA MUST be updated with fix version
Are there any plans to backport to branch-3.3? A retest of the cherrypick is all which should be needed

Keeping both branches in sync is essential for cherrypicking future work

surendralilhore · 2021-06-09T05:11:29Z

@steveloughran , I updated the Jira.

Are there any plans to backport to branch-3.3? A retest of the cherrypick is all which should be needed

Yes, @sumangala-patki is working on backport and testing it for 3.2 and 3.3 branch.

sumangala-patki · 2021-06-09T05:35:52Z

@surendralilhore thanks for updating the JIRA
@steveloughran yes, but planning to ensure a gap (maybe a week or two) between check-in to trunk and backport of PRs. Will keep a track of pending backports

…cessors) to const (apache#2795) . Contributed by Sumangala Patki. (cherry picked from commit 76d92eb)

…cessors) to const (apache#2795) . Contributed by Sumangala Patki.

…cessors) to const (#3106) * HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const (#2795) . Contributed by Sumangala Patki. (cherry picked from commit 76d92eb)

…cessors) to const (apache#2795) . Contributed by Sumangala Patki.

sumangala17 added 5 commits March 22, 2021 16:48

set default to 2

be54ae5

test

b7d1366

fix test

1dfd213

rm import diff

3320a12

import

8bf5862

Merge branch 'trunk' into HADOOP-17596

b165323

sumangala-patki marked this pull request as ready for review March 24, 2021 11:09

doc

b8fc345

vinaysbadami approved these changes Mar 26, 2021

View reviewed changes

bilaharith approved these changes Mar 27, 2021

View reviewed changes

snvijaya approved these changes Mar 27, 2021

View reviewed changes

anoopsjohn reviewed May 13, 2021

View reviewed changes

...s/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsInputStream.java Outdated Show resolved Hide resolved

sumangala17 added 2 commits May 13, 2021 15:38

const

6611791

Merge branch 'trunk' into HADOOP-17596

56635ba

surendralilhore approved these changes Jun 3, 2021

View reviewed changes

surendralilhore merged commit 76d92eb into apache:trunk Jun 3, 2021

sumangala-patki added a commit to sumangala17/hadoop that referenced this pull request Jun 15, 2021

HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(pro…

e888bb3

…cessors) to const (apache#2795) . Contributed by Sumangala Patki. (cherry picked from commit 76d92eb)

bilaharith pushed a commit to bilaharith/hadoop that referenced this pull request Jun 18, 2021

HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(pro…

a8edc3b

…cessors) to const (apache#2795) . Contributed by Sumangala Patki.

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021

HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(pro…

4d966d6

…cessors) to const (apache#2795) . Contributed by Sumangala Patki.

HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const #2795

HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const #2795

Uh oh!

Conversation

sumangala-patki commented Mar 22, 2021

Uh oh!

hadoop-yetus commented Mar 22, 2021

Uh oh!

hadoop-yetus commented Mar 23, 2021

Uh oh!

sumangala-patki commented Mar 23, 2021

Uh oh!

hadoop-yetus commented Mar 26, 2021

Uh oh!

snvijaya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

anoopsjohn commented May 13, 2021

Uh oh!

hadoop-yetus commented May 13, 2021

Uh oh!

vinaysbadami commented May 17, 2021

Uh oh!

anoopsjohn commented May 17, 2021

Uh oh!

anoopsjohn commented May 17, 2021

Uh oh!

surendralilhore left a comment

Choose a reason for hiding this comment

Uh oh!

surendralilhore commented Jun 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steveloughran commented Jun 8, 2021

Uh oh!

surendralilhore commented Jun 9, 2021

Uh oh!

sumangala-patki commented Jun 9, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

surendralilhore commented Jun 3, 2021 •

edited

Loading