HADOOP-18180. Replaces ExecutorServicePool with ExecutorService #4520

ahmarsuhail · 2022-06-30T11:47:57Z

Description of PR

The ExecutorServicePool class was only submitting tasks to the ExecutorService. This PR removes ExecutorServicePool and submits directly to the ExecutorService.

How was this patch tested?

Tested in eu-west-1 by running

mvn -Dparallel-tests -DtestsThreadCount=16 clean verify

This is the the initial merge of the HADOOP-18028 S3A performance input stream. This patch on its own is incomplete and must be accompanied by all other commits with HADOOP-18028 in their git commit message. Consult the JIRA for that list Contributed by Bhalchandra Pandit.

…3A prefetching stream (apache#4115) Contributed by PJ Fanning.

Contributed by Ahmar Suhail

…ache#4212) Contributed by Monthon Klongklaew

dannycjones

I have a concern on which thread pool should be passed by S3AFileSystem; otherwise, looking good.

dannycjones · 2022-06-30T12:08:30Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

-        futurePool,
+        unboundedThreadPool,


Why not the boundedThreadPool originally wrapped by futurePool?

that's my bad, it should be boundedThreadPool. Have updated

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AReadOpContext.java

dannycjones

+1 (non-binding) assuming Yetus is happy

hadoop-yetus · 2022-06-30T13:41:51Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	13m 8s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+1 💚	mvninstall	38m 42s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	1m 2s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	0m 56s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 49s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	1m 1s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	0m 50s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 52s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 38s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	22m 0s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 44s		the patch passed
+1 💚	compile	0m 48s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	0m 48s		the patch passed
+1 💚	compile	0m 39s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 39s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 26s		hadoop-tools/hadoop-aws: The patch generated 0 new + 3 unchanged - 2 fixed = 3 total (was 5)
+1 💚	mvnsite	0m 43s		the patch passed
+1 💚	javadoc	0m 23s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 35s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 12s		the patch passed
+1 💚	shadedclient	21m 17s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	2m 40s	/patch-unit-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch passed.
+1 💚	asflicense	0m 49s		The patch does not generate ASF License warnings.
		112m 28s

Reason	Tests
Failed junit tests	hadoop.fs.s3a.read.TestS3InputStream

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4520/1/artifact/out/Dockerfile
GITHUB PR	#4520
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux caff41b93769 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `225f0f0`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4520/1/testReport/
Max. process+thread count	716 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4520/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

ahmarsuhail · 2022-06-30T13:43:16Z

Failing unit tests are fixed in #4386

hadoop-yetus · 2022-06-30T14:31:58Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	17m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+1 💚	mvninstall	40m 41s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	0m 55s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	0m 47s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 42s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	0m 55s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	0m 40s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 42s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 29s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	23m 40s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 38s		the patch passed
+1 💚	compile	0m 44s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	0m 44s		the patch passed
+1 💚	compile	0m 34s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 34s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 23s		hadoop-tools/hadoop-aws: The patch generated 0 new + 3 unchanged - 2 fixed = 3 total (was 5)
+1 💚	mvnsite	0m 40s		the patch passed
+1 💚	javadoc	0m 21s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 30s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 15s		the patch passed
+1 💚	shadedclient	23m 31s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	2m 41s	/patch-unit-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch passed.
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		120m 53s

Reason	Tests
Failed junit tests	hadoop.fs.s3a.read.TestS3InputStream

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4520/2/artifact/out/Dockerfile
GITHUB PR	#4520
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 8eb241625259 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `3bd7b73`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4520/2/testReport/
Max. process+thread count	566 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4520/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

apache#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail

Contributed by Ahmar Suhail <[email protected]>

This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail

…move-future-pool

hadoop-yetus · 2022-07-26T15:13:35Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	15m 17s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
-1 ❌	mvninstall	0m 32s	/branch-mvninstall-root.txt	root in feature-HADOOP-18028-s3a-prefetch failed.
-1 ❌	compile	0m 31s	/branch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt	hadoop-aws in feature-HADOOP-18028-s3a-prefetch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.
-1 ❌	compile	0m 32s	/branch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt	hadoop-aws in feature-HADOOP-18028-s3a-prefetch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.
-0 ⚠️	checkstyle	0m 30s	/buildtool-branch-checkstyle-hadoop-tools_hadoop-aws.txt	The patch fails to run checkstyle in hadoop-aws
-1 ❌	mvnsite	0m 32s	/branch-mvnsite-hadoop-tools_hadoop-aws.txt	hadoop-aws in feature-HADOOP-18028-s3a-prefetch failed.
-1 ❌	javadoc	0m 31s	/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt	hadoop-aws in feature-HADOOP-18028-s3a-prefetch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.
-1 ❌	javadoc	0m 31s	/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt	hadoop-aws in feature-HADOOP-18028-s3a-prefetch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.
-1 ❌	spotbugs	0m 31s	/branch-spotbugs-hadoop-tools_hadoop-aws.txt	hadoop-aws in feature-HADOOP-18028-s3a-prefetch failed.
+1 💚	shadedclient	3m 37s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	4m 9s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
-1 ❌	mvninstall	0m 55s	/patch-mvninstall-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
-1 ❌	compile	4m 0s	/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt	hadoop-aws in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.
-1 ❌	javac	4m 0s	/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt	hadoop-aws in the patch failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.
-1 ❌	compile	1m 36s	/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt	hadoop-aws in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.
-1 ❌	javac	1m 36s	/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt	hadoop-aws in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 29s	/buildtool-patch-checkstyle-hadoop-tools_hadoop-aws.txt	The patch fails to run checkstyle in hadoop-aws
-1 ❌	mvnsite	1m 7s	/patch-mvnsite-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
+1 💚	javadoc	0m 42s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
-1 ❌	javadoc	0m 38s	/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt	hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 generated 40 new + 0 unchanged - 0 fixed = 40 total (was 0)
-1 ❌	spotbugs	0m 31s	/patch-spotbugs-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
-1 ❌	shadedclient	6m 52s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	0m 30s	/patch-unit-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
+0 🆗	asflicense	0m 31s		ASF License check generated no output?
		36m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4520/3/artifact/out/Dockerfile
GITHUB PR	#4520
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 7d5b7ab1478f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `d3f6d6b`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4520/3/testReport/
Max. process+thread count	88 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4520/3/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-07-26T17:15:28Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 43s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+1 💚	mvninstall	41m 17s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	1m 10s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	1m 1s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 56s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	1m 14s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	0m 50s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 56s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 56s		feature-HADOOP-18028-s3a-prefetch passed
-1 ❌	shadedclient	29m 48s		branch has errors when building and testing our client artifacts.
-0 ⚠️	patch	30m 27s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
-1 ❌	mvninstall	0m 31s	/patch-mvninstall-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
+1 💚	compile	0m 53s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	0m 53s		the patch passed
+1 💚	compile	0m 52s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 52s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 30s	/buildtool-patch-checkstyle-hadoop-tools_hadoop-aws.txt	The patch fails to run checkstyle in hadoop-aws
-1 ❌	mvnsite	0m 29s	/patch-mvnsite-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
+1 💚	javadoc	0m 26s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 35s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 36s		the patch passed
-1 ❌	shadedclient	9m 44s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	0m 41s	/patch-unit-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
+0 🆗	asflicense	0m 38s		ASF License check generated no output?
		97m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4520/4/artifact/out/Dockerfile
GITHUB PR	#4520
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 45ab6f2bb7f9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `9ddad61`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4520/4/testReport/
Max. process+thread count	411 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4520/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran and others added 6 commits May 19, 2022 12:52

HADOOP-18180. Replace use of twitter util-core with java futures in S…

9ebfcce

…3A prefetching stream (apache#4115) Contributed by PJ Fanning.

HADOOP-18177. Document prefetching architecture. (apache#4205)

b653cfa

Contributed by Ahmar Suhail

HADOOP-18175. fix test failures with prefetching s3a input stream (ap…

16d5403

…ache#4212) Contributed by Monthon Klongklaew

fixes compilation errors in tests

b75b72b

replaces ExecutorServiceFuturePool with ExecutorService

225f0f0

dannycjones suggested changes Jun 30, 2022

View reviewed changes

use boundedThreadPool

3bd7b73

dannycjones approved these changes Jun 30, 2022

View reviewed changes

ahmarsuhail and others added 4 commits July 15, 2022 13:51

HADOOP-18254. Disable S3A prefetching by default. (apache#4469)

dc09c61

Contributed by Ahmar Suhail <[email protected]>

HADOOP-18190. Collect IOStatistics during S3A prefetching (apache#4458)

a964fbf

This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail

Merge branch 'feature-HADOOP-18028-s3a-prefetch' into HADOOP-18180-re…

d3f6d6b

…move-future-pool

rerun yetus

9ddad61

asfgit force-pushed the feature-HADOOP-18028-s3a-prefetch branch from a964fbf to e23f70a Compare August 3, 2022 12:46

ahmarsuhail closed this Aug 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-18180. Replaces ExecutorServicePool with ExecutorService #4520

HADOOP-18180. Replaces ExecutorServicePool with ExecutorService #4520

Uh oh!

ahmarsuhail commented Jun 30, 2022

Uh oh!

dannycjones left a comment

Uh oh!

dannycjones Jun 30, 2022

Uh oh!

ahmarsuhail Jun 30, 2022

Uh oh!

Uh oh!

dannycjones left a comment

Uh oh!

hadoop-yetus commented Jun 30, 2022

Uh oh!

ahmarsuhail commented Jun 30, 2022

Uh oh!

hadoop-yetus commented Jun 30, 2022

Uh oh!

hadoop-yetus commented Jul 26, 2022

Uh oh!

hadoop-yetus commented Jul 26, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

HADOOP-18180. Replaces ExecutorServicePool with ExecutorService #4520

HADOOP-18180. Replaces ExecutorServicePool with ExecutorService #4520

Uh oh!

Conversation

ahmarsuhail commented Jun 30, 2022

Description of PR

How was this patch tested?

Uh oh!

dannycjones left a comment

Choose a reason for hiding this comment

Uh oh!

dannycjones Jun 30, 2022

Choose a reason for hiding this comment

Uh oh!

ahmarsuhail Jun 30, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dannycjones left a comment

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Jun 30, 2022

Uh oh!

ahmarsuhail commented Jun 30, 2022

Uh oh!

hadoop-yetus commented Jun 30, 2022

Uh oh!

hadoop-yetus commented Jul 26, 2022

Uh oh!

hadoop-yetus commented Jul 26, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants