HADOOP-18190. Adds iostats for prefetching #4458

ahmarsuhail · 2022-06-18T13:39:39Z

Description of PR

This PR adds in iostats for the prefetching input stream. This PR is dependent on #4386, which fixes issues after the rebase. Once that gets in merged in, this PR can also update ITestS3PrefetchingInputStream with assertions on more stats.

The following stats are added:

Counters:

STREAM_READ_PREFETCH_OPERATIONS: Total number of prefetch ops requested

Duration:

STREAM_READ_REMOTE_BLOCK_READ: Time taken to read a full block of data from S3
STREAM_READ_BLOCK_ACQUIRE_AND_READ: Time taken to acquire a buffer from the pool and read data into it. This is
not for prefetching ops, but when the first block of data is read (on first read or after a seek)
ACTION_EXECUTOR_ACQUIRED: Time taken to acquire an executor. Either by the prefetching task or the caching task.

Gauges:

STREAM_READ_BLOCKS_IN_FILE_CACHE: Blocks currently in file cache
STREAM_READ_ACTIVE_PREFETCH_OPERATIONS: Current active prefetch operations
STREAM_READ_ACTIVE_MEMORY_IN_USE: Current active memory in use

There are still some more stats that can be added, especially around cache usage. I'll follow up with another PR for those if we're happy with the approach of this PR.

How was this patch tested?

Tested in eu-west-1 by running

mvn -Dparallel-tests -DtestsThreadCount=16 clean verify

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java

monthonk

The approach looks good to me, suggesting some changes.

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/BufferPool.java

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/CachingBlockManager.java

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/SingleFilePerBlockCache.java

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInstrumentation.java

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java

...op-aws/src/main/java/org/apache/hadoop/fs/s3a/statistics/impl/EmptyS3AStatisticsContext.java

monthonk

LGTM

steveloughran

This all looks great!

As you said:

Discussed offline, I think we should move them to this class StreamStatisticNames, but let's see what other people think.l

please.

testing

For the tests, you will need to isolate the classes and tests we want to move to common, so add an empty PrefetchingStatistics implementation there.

It'd be really useful to actually make assertions on the stats so that you can verify that

stats are collected in the stream.
after the stream is closed, the stats are visible in the FileSystem IOStatistics. Use IOStatisticAssertions here

In #4352 @mehakmeet is adding thread-level stats collection in the input and output streams; we are going to get spark to collect these in the workers and report to the spark driver. This will give you aggregate stats of most store IO from the entire job. This is going to be awesome. The stats being collected here are critical for this -we will have to make sure that they get pushed to the thread context stats of the thread opening the file. Don't worry about that yet

steveloughran · 2022-07-15T13:02:52Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java

+  /**
+   * Total number of prefetching operations executed.
+   */
+  public static final String STREAM_READ_PREFETCH_OPERATIONS


as you discussed, move to StreamStatisticNames in hadoop-common

steveloughran · 2022-07-15T13:05:23Z

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/TestBlockCache.java

    // Should not throw.
-    BlockCache cache = new SingleFilePerBlockCache();
+    BlockCache cache =
+        new SingleFilePerBlockCache(EmptyS3AStatisticsContext.EMPTY_INPUT_STREAM_STATISTICS);


when we move to hadoop common the s3a classes won't be available. Add an empty implementation in this test package and use it

…efetching-add-iostats

hadoop-yetus · 2022-07-18T12:15:07Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 42s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+1 💚	mvninstall	37m 8s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	1m 3s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	0m 56s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 52s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	1m 4s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	0m 50s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 53s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 36s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	21m 1s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	21m 28s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 42s		the patch passed
+1 💚	compile	0m 45s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	0m 45s		the patch passed
+1 💚	compile	0m 39s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 39s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 28s		the patch passed
+1 💚	mvnsite	0m 43s		the patch passed
+1 💚	javadoc	0m 26s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 32s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 16s		the patch passed
+1 💚	shadedclient	20m 29s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 46s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 51s		The patch does not generate ASF License warnings.
		109m 1s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/4/artifact/out/Dockerfile
GITHUB PR	#4458
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 635425a9b1e4 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `65f5ee7`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/4/testReport/
Max. process+thread count	616 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-07-18T17:13:08Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+0 🆗	mvndep	14m 31s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 15s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	23m 18s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	20m 47s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 24s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	3m 45s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	3m 3s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 44s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 5s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	22m 16s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	22m 49s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 34s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 48s		the patch passed
+1 💚	compile	22m 25s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	22m 25s		the patch passed
+1 💚	compile	20m 34s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	20m 34s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 12s		the patch passed
+1 💚	mvnsite	3m 45s		the patch passed
+1 💚	javadoc	2m 54s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 42s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 12s		the patch passed
+1 💚	shadedclient	22m 17s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 32s		hadoop-common in the patch passed.
+1 💚	unit	3m 15s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 38s		The patch does not generate ASF License warnings.
		240m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/5/artifact/out/Dockerfile
GITHUB PR	#4458
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 3600ed867d00 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `becdd66`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/5/testReport/
Max. process+thread count	2756 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran

looking good. I am now worried about how errors during read are (a) logged and (b) measured in the stats.

proposed

errors in prefetch only logged at info with error text but not stack
full stack logged at debug

but: we do want the most recent failure to be raised on the next read() on the stream when there is no data in the cache.

something for a separate PR, but think about it.

for this one, can you split success/failure logging of the invocation and duration of calls, helps identify

cases where failures happen after long delays compared to success
cases where failures are immediate

steveloughran · 2022-07-18T17:54:45Z

...roject/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/StreamStatisticNames.java

+
+  /**
+   * count/duration of reading a remote block.
+   * IO.


why the IO here and below?

steveloughran · 2022-07-18T17:57:44Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/BufferPool.java

   * @throws IllegalArgumentException if bufferSize is zero or negative.
   */
-  public BufferPool(int size, int bufferSize) {
+  public BufferPool(int size, int bufferSize, PrefetchingStatistics prefetchingStatistics) {


we are going to have to rename this class; org.apache.hadoop.io.ByteBufferPool has long existed and it will only cause confusion. not in this PR, but soon

steveloughran · 2022-07-18T17:59:52Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/CachingBlockManager.java

        buffer.flip();
        data.setReady(expectedState);
+
+        if(isPrefetch) {


nit, spacing

steveloughran · 2022-07-18T18:00:42Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/BufferPool.java

      }
    }

+    int currentPoolSize = this.pool.numCreated();


how about you cut the this. here and from the existing calls in the same method. not our project's style

steveloughran · 2022-07-18T18:03:07Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/CachingBlockManager.java

-        this.blockManager.prefetch(data);
+        this.blockManager.prefetch(data, taskQueuedStartTime);
      } catch (Exception e) {
        LOG.error("error during prefetch", e);


i worry about this. seen problems with transient network failures and abfs where the logs are flooded with many parallel stack traces before it recovers.

steveloughran · 2022-07-18T18:07:59Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/CachingBlockManager.java

        data.setReady(expectedState);
+
+        if(isPrefetch) {
+          this.prefetchingStatistics.prefetchOperationCompleted();


what happens if the read failed? is the failure count and duration before failure recorded separately from the successful calls?

So this was currently only measuring successful calls, I'm now also adding a count for failure. I can't figure out how to measure duration for a failure though, could you point me to an example?

ahmarsuhail · 2022-07-19T16:24:41Z

@steveloughran by

can you split success/failure logging of the invocation and duration of calls

do you mean to add in stats for number of failed prefetch ops & duration of this failure? for duration, I couldn't figure out how to measure failure..for example, the duration of reading from S3 is measured here, if the operation fails, the duration tracker will call tracker.failed();.

What does tracker.failed() do?
how should this be changed to measure duration of a failure?

…3PrefetchingInputStream

hadoop-yetus · 2022-07-20T14:41:11Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 48s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+0 🆗	mvndep	14m 48s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 18s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	23m 3s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	21m 27s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 22s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	3m 20s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	2m 33s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 2s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	4m 42s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	23m 3s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	23m 32s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 50s		the patch passed
+1 💚	compile	24m 41s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	24m 41s		the patch passed
+1 💚	compile	22m 14s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	22m 14s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 6s		the patch passed
+1 💚	mvnsite	3m 26s		the patch passed
+1 💚	javadoc	2m 41s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 32s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 7s		the patch passed
+1 💚	shadedclient	22m 32s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 45s		hadoop-common in the patch passed.
+1 💚	unit	3m 7s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 30s		The patch does not generate ASF License warnings.
		254m 45s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/6/artifact/out/Dockerfile
GITHUB PR	#4458
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 20d8139f0c38 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `1d710f1`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/6/testReport/
Max. process+thread count	1349 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-07-20T14:54:41Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	17m 31s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+0 🆗	mvndep	14m 33s		Maven dependency ordering for branch
+1 💚	mvninstall	27m 59s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	24m 57s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	21m 41s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 27s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	3m 12s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	2m 23s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 5s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	4m 40s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	24m 37s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	25m 5s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 25s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 43s		the patch passed
+1 💚	compile	24m 12s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	24m 12s		the patch passed
+1 💚	compile	21m 42s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	21m 42s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 21s		the patch passed
+1 💚	mvnsite	3m 12s		the patch passed
+1 💚	javadoc	2m 15s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 5s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	4m 51s		the patch passed
+1 💚	shadedclient	25m 7s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 13s		hadoop-common in the patch passed.
+1 💚	unit	3m 7s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 16s		The patch does not generate ASF License warnings.
		264m 26s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/7/artifact/out/Dockerfile
GITHUB PR	#4458
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 195ef8bf1a6b 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `1d710f1`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/7/testReport/
Max. process+thread count	1252 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/7/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran · 2022-07-20T14:57:56Z

look at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding#trackFunctionDuration for an example of failure handling. the failed() call switches to the same stats key with the suffix .failures and updates those counters and durations

steveloughran

reviewed the changes. hope I've clarified how DurationTracker automatically uses the .failures stats if told of a failure.

steveloughran · 2022-07-20T15:00:29Z

...roject/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/StreamStatisticNames.java

+  /**
+   * Total number of failed prefetching operations.
+   */
+  public static final String STREAM_READ_FAILED_PREFETCH_OPERATIONS


not needed. every duration type you build for a store automatically gets .failures stats entries (count, min, mean, max), use StoreStatisticNames.SUFFIX_FAILURES if you want to see usages

steveloughran · 2022-07-20T15:59:46Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3Reader.java

-        "read", this.s3File.getPath(), true,
-        () -> {
+    int invokerResponse = invoker.retry("read", this.s3File.getPath(), true,
+        trackDurationOfOperation(streamStatistics, STREAM_READ_REMOTE_BLOCK_READ, () -> {


FYI, this will do the right thing about updating the stream statistics with the pass/fail stats differentiated

steveloughran · 2022-07-20T16:03:00Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/CachingBlockManager.java

        }

        if (isPrefetch) {
+          prefetchingStatistics.prefetchOperationStarted();


have this return a DurationTracker whose close() will update the statistic. on an an exception, call its failed() method to update the .failure keys intead

steveloughran · 2022-07-20T16:04:15Z

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/PrefetchingStatistics.java

+
+import org.apache.hadoop.fs.statistics.IOStatisticsSource;
+
+public interface PrefetchingStatistics extends IOStatisticsSource {


needs javadocs i'm afraid

...p-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java

steveloughran · 2022-07-20T16:09:39Z

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/TestBufferPool.java

        IllegalArgumentException.class,
        "'size' must be a positive integer",
-        () -> new BufferPool(0, 10));
+        () -> new BufferPool(0, 10, EmptyS3AStatisticsContext.EMPTY_INPUT_STREAM_STATISTICS));


use newInputStreamStatistics() which is part of S3AStatisticsContext.

hadoop-yetus · 2022-07-21T19:38:49Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	21m 2s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+0 🆗	mvndep	14m 45s		Maven dependency ordering for branch
+1 💚	mvninstall	28m 20s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	26m 41s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	23m 53s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 45s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	4m 29s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	2m 39s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 14s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 16s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	26m 17s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	26m 45s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 32s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 48s		the patch passed
+1 💚	compile	27m 28s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	27m 28s		the patch passed
+1 💚	compile	23m 2s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	23m 2s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 43s	/results-checkstyle-root.txt	root: The patch generated 8 new + 1 unchanged - 0 fixed = 9 total (was 1)
+1 💚	mvnsite	4m 23s		the patch passed
+1 💚	javadoc	2m 24s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 13s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
-1 ❌	spotbugs	1m 53s	/new-spotbugs-hadoop-tools_hadoop-aws.html	hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	27m 19s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	20m 44s		hadoop-common in the patch passed.
+1 💚	unit	3m 22s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 19s		The patch does not generate ASF License warnings.
		288m 57s

Reason	Tests
SpotBugs	module:hadoop-tools/hadoop-aws
	Possible null pointer dereference of tracker in org.apache.hadoop.fs.common.CachingBlockManager.readBlock(BufferData, boolean, BufferData$State[]) Dereferenced at CachingBlockManager.java:tracker in org.apache.hadoop.fs.common.CachingBlockManager.readBlock(BufferData, boolean, BufferData$State[]) Dereferenced at CachingBlockManager.java:[line 365]

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/8/artifact/out/Dockerfile
GITHUB PR	#4458
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 2beff22d3a61 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `66fb591`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/8/testReport/
Max. process+thread count	1247 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/8/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-07-21T20:23:21Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 26s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+0 🆗	mvndep	15m 17s		Maven dependency ordering for branch
+1 💚	mvninstall	31m 49s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	28m 0s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	23m 31s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	5m 26s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	3m 34s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	2m 37s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 10s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 5s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	26m 4s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	26m 34s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 2s		the patch passed
+1 💚	compile	26m 54s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	26m 54s		the patch passed
+1 💚	compile	22m 43s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	22m 43s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 31s	/results-checkstyle-root.txt	root: The patch generated 8 new + 1 unchanged - 0 fixed = 9 total (was 1)
+1 💚	mvnsite	3m 27s		the patch passed
+1 💚	javadoc	2m 21s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 7s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
-1 ❌	spotbugs	2m 8s	/new-spotbugs-hadoop-tools_hadoop-aws.html	hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	25m 30s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 14s		hadoop-common in the patch passed.
+1 💚	unit	3m 12s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 18s		The patch does not generate ASF License warnings.
		266m 57s

Reason	Tests
SpotBugs	module:hadoop-tools/hadoop-aws
	Possible null pointer dereference of tracker in org.apache.hadoop.fs.common.CachingBlockManager.readBlock(BufferData, boolean, BufferData$State[]) Dereferenced at CachingBlockManager.java:tracker in org.apache.hadoop.fs.common.CachingBlockManager.readBlock(BufferData, boolean, BufferData$State[]) Dereferenced at CachingBlockManager.java:[line 365]

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/9/artifact/out/Dockerfile
GITHUB PR	#4458
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 716ab780773f 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `62d1b8d`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/9/testReport/
Max. process+thread count	1252 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/9/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-07-22T12:42:04Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+0 🆗	mvndep	14m 41s		Maven dependency ordering for branch
+1 💚	mvninstall	27m 37s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	24m 54s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	22m 35s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 20s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	3m 26s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	2m 41s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 17s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 23s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	23m 18s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	23m 51s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 52s		the patch passed
+1 💚	compile	24m 16s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	24m 16s		the patch passed
+1 💚	compile	21m 54s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	21m 54s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 23s	/results-checkstyle-root.txt	root: The patch generated 23 new + 0 unchanged - 1 fixed = 23 total (was 1)
+1 💚	mvnsite	3m 39s		the patch passed
+1 💚	javadoc	2m 53s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 44s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
-1 ❌	spotbugs	2m 6s	/new-spotbugs-hadoop-tools_hadoop-aws.html	hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	22m 26s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 40s		hadoop-common in the patch passed.
+1 💚	unit	3m 9s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 24s		The patch does not generate ASF License warnings.
		250m 26s

Reason	Tests
SpotBugs	module:hadoop-tools/hadoop-aws
	Possible null pointer dereference of tracker in org.apache.hadoop.fs.common.CachingBlockManager.readBlock(BufferData, boolean, BufferData$State[]) Dereferenced at CachingBlockManager.java:tracker in org.apache.hadoop.fs.common.CachingBlockManager.readBlock(BufferData, boolean, BufferData$State[]) Dereferenced at CachingBlockManager.java:[line 365]

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/10/artifact/out/Dockerfile
GITHUB PR	#4458
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 3c0df40fe439 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `5834ea9`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/10/testReport/
Max. process+thread count	1251 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/10/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-07-22T13:02:44Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 10s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+0 🆗	mvndep	14m 28s		Maven dependency ordering for branch
+1 💚	mvninstall	28m 9s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	25m 19s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	21m 46s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 35s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	3m 18s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	2m 27s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 9s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	4m 44s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	25m 0s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	25m 27s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 25s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 43s		the patch passed
+1 💚	compile	24m 23s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	24m 23s		the patch passed
+1 💚	compile	21m 51s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	21m 51s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 19s	/results-checkstyle-root.txt	root: The patch generated 23 new + 0 unchanged - 1 fixed = 23 total (was 1)
+1 💚	mvnsite	3m 12s		the patch passed
+1 💚	javadoc	2m 17s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 9s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	4m 53s		the patch passed
+1 💚	shadedclient	24m 42s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 11s		hadoop-common in the patch passed.
+1 💚	unit	3m 5s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 19s		The patch does not generate ASF License warnings.
		249m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/11/artifact/out/Dockerfile
GITHUB PR	#4458
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux ab6b4456d883 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `e556425`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/11/testReport/
Max. process+thread count	1252 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/11/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran · 2022-07-22T19:03:10Z

not sure why one of the runs didn't pick up the tracker null checks. looks like there's an indentation to fix in one file

./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:28: public final class EmptyPrefetchingStatistics implements PrefetchingStatistics {: 'class def modifier' has incorrect indentation level 1, expected level should be 0. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:30:   private static final EmptyPrefetchingStatistics EMPTY_PREFETCHING_STATISTICS =: 'member def modifier' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:33:   private EmptyPrefetchingStatistics() {: 'ctor def modifier' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:34:   }: 'ctor def rcurly' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:36:   public static EmptyPrefetchingStatistics getInstance() {: 'method def modifier' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:37:     return EMPTY_PREFETCHING_STATISTICS;: 'method def' child has incorrect indentation level 5, expected level should be 4. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:38:   }: 'method def rcurly' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:40:   @Override: 'method def modifier' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:42:     return stubDurationTracker();: 'method def' child has incorrect indentation level 5, expected level should be 4. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:43:   }: 'method def rcurly' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:45:   @Override: 'method def modifier' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:48:   }: 'method def rcurly' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:50:   @Override: 'method def modifier' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:53:   }: 'method def rcurly' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:55:   @Override: 'method def modifier' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:58:   }: 'method def rcurly' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:60:   @Override: 'method def modifier' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:63:   }: 'method def rcurly' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:65:   @Override: 'method def modifier' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:68:   }: 'method def rcurly' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:70:   @Override: 'method def modifier' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:73:   }: 'method def rcurly' has incorrect indentation level 3, expected level should be 2. [Indentation]
./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:74: }: 'class def rcurly' has incorrect indentation level 1, expected level should be 0. [Indentation]

i think once those are done we can declare victory and merge, though the stream will need to support IOStatisticsContext after the next rebase

steveloughran

+1 if checkstyle is happy.

hadoop-yetus · 2022-07-25T18:02:33Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 47s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+0 🆗	mvndep	15m 7s		Maven dependency ordering for branch
+1 💚	mvninstall	28m 27s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	26m 39s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	23m 7s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 32s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	3m 29s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	2m 44s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 21s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 7s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	25m 0s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	25m 32s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 46s		the patch passed
+1 💚	compile	24m 31s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	24m 31s		the patch passed
+1 💚	compile	22m 23s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	22m 23s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 12s	/results-checkstyle-root.txt	root: The patch generated 1 new + 0 unchanged - 1 fixed = 1 total (was 1)
+1 💚	mvnsite	3m 33s		the patch passed
+1 💚	javadoc	2m 29s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 19s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 25s		the patch passed
+1 💚	shadedclient	23m 18s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 49s		hadoop-common in the patch passed.
+1 💚	unit	3m 11s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 24s		The patch does not generate ASF License warnings.
		255m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/12/artifact/out/Dockerfile
GITHUB PR	#4458
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux feb2707df24c 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `492b2ef`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/12/testReport/
Max. process+thread count	1251 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/12/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-07-25T23:31:07Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 43s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ feature-HADOOP-18028-s3a-prefetch Compile Tests _
+0 🆗	mvndep	14m 54s		Maven dependency ordering for branch
+1 💚	mvninstall	26m 24s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	compile	24m 57s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	22m 0s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 38s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	mvnsite	3m 33s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	javadoc	3m 2s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 44s		feature-HADOOP-18028-s3a-prefetch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 7s		feature-HADOOP-18028-s3a-prefetch passed
+1 💚	shadedclient	23m 33s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	24m 6s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 47s		the patch passed
+1 💚	compile	24m 18s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	24m 18s		the patch passed
+1 💚	compile	21m 57s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	21m 57s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 15s		root: The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
+1 💚	mvnsite	3m 40s		the patch passed
+1 💚	javadoc	2m 42s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	2m 25s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	5m 18s		the patch passed
+1 💚	shadedclient	25m 13s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 58s		hadoop-common in the patch passed.
+1 💚	unit	3m 19s		hadoop-aws in the patch passed.
+1 💚	asflicense	1m 32s		The patch does not generate ASF License warnings.
		252m 12s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/13/artifact/out/Dockerfile
GITHUB PR	#4458
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 24427805232f 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	feature-HADOOP-18028-s3a-prefetch / `021c98a`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/13/testReport/
Max. process+thread count	1251 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/13/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

ahmarsuhail · 2022-07-26T07:44:21Z

thanks for reviewing @steveloughran, looks like checkstyle is happy now

steveloughran · 2022-07-26T11:01:46Z

+1, merging

This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail

This is the the a rollup patch of the HADOOP-18028 S3A performance input stream feature branch. Contains HADOOP-18028. High performance S3A input stream (apache#4109) This is the the merge of the HADOOP-18028 S3A performance input stream. This patch on its own is incomplete and must be accompanied by all other commits with HADOOP-18028 in their git commit message. Consult the JIRA for that list Contributed by Bhalchandra Pandit. HADOOP-18180. Replace use of twitter util-core with java futures in S3A prefetching stream (apache#4115) Contributed by PJ Fanning. HADOOP-18177. Document prefetching architecture. (apache#4205) Contributed by Ahmar Suhail HADOOP-18175. fix test failures with prefetching s3a input stream (apache#4212) Contributed by Monthon Klongklaew HADOOP-18231. S3A prefetching: fix failing tests & drain stream async. (apache#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail HADOOP-18254. Disable S3A prefetching by default. (apache#4469) Contributed by Ahmar Suhail HADOOP-18190. Collect IOStatistics during S3A prefetching (apache#4458) This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail. Change-Id: I48f217086531c12d6e2f0f91e39f17054a74d20f

This is the the preview release of the HADOOP-18028 S3A performance input stream. It is still stabilizing, but ready to test. Contains HADOOP-18028. High performance S3A input stream (apache#4109) Contributed by Bhalchandra Pandit. HADOOP-18180. Replace use of twitter util-core with java futures (apache#4115) Contributed by PJ Fanning. HADOOP-18177. Document prefetching architecture. (apache#4205) Contributed by Ahmar Suhail HADOOP-18175. fix test failures with prefetching s3a input stream (apache#4212) Contributed by Monthon Klongklaew HADOOP-18231. S3A prefetching: fix failing tests & drain stream async. (apache#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail HADOOP-18254. Disable S3A prefetching by default. (apache#4469) Contributed by Ahmar Suhail HADOOP-18190. Collect IOStatistics during S3A prefetching (apache#4458) This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail HADOOP-18379 rebase feature/HADOOP-18028-s3a-prefetch to trunk HADOOP-18187. Convert s3a prefetching to use JavaDoc for fields and enums. HADOOP-18318. Update class names to be clear they belong to S3A prefetching Contributed by Steve Loughran Change-Id: I6511c51c3580c57eb72e8ea686c88e3917d12a06

This is the the preview release of the HADOOP-18028 S3A performance input stream. It is still stabilizing, but ready to test. Contains HADOOP-18028. High performance S3A input stream (#4109) Contributed by Bhalchandra Pandit. HADOOP-18180. Replace use of twitter util-core with java futures (#4115) Contributed by PJ Fanning. HADOOP-18177. Document prefetching architecture. (#4205) Contributed by Ahmar Suhail HADOOP-18175. fix test failures with prefetching s3a input stream (#4212) Contributed by Monthon Klongklaew HADOOP-18231. S3A prefetching: fix failing tests & drain stream async. (#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail HADOOP-18254. Disable S3A prefetching by default. (#4469) Contributed by Ahmar Suhail HADOOP-18190. Collect IOStatistics during S3A prefetching (#4458) This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail HADOOP-18379 rebase feature/HADOOP-18028-s3a-prefetch to trunk HADOOP-18187. Convert s3a prefetching to use JavaDoc for fields and enums. HADOOP-18318. Update class names to be clear they belong to S3A prefetching Contributed by Steve Loughran

This is the the preview release of the HADOOP-18028 S3A performance input stream. It is still stabilizing, but ready to test. Contains HADOOP-18028. High performance S3A input stream (apache#4109) Contributed by Bhalchandra Pandit. HADOOP-18180. Replace use of twitter util-core with java futures (apache#4115) Contributed by PJ Fanning. HADOOP-18177. Document prefetching architecture. (apache#4205) Contributed by Ahmar Suhail HADOOP-18175. fix test failures with prefetching s3a input stream (apache#4212) Contributed by Monthon Klongklaew HADOOP-18231. S3A prefetching: fix failing tests & drain stream async. (apache#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail HADOOP-18254. Disable S3A prefetching by default. (apache#4469) Contributed by Ahmar Suhail HADOOP-18190. Collect IOStatistics during S3A prefetching (apache#4458) This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail HADOOP-18379 rebase feature/HADOOP-18028-s3a-prefetch to trunk HADOOP-18187. Convert s3a prefetching to use JavaDoc for fields and enums. HADOOP-18318. Update class names to be clear they belong to S3A prefetching Contributed by Steve Loughran

This is the the preview release of the HADOOP-18028 S3A performance input stream. It is still stabilizing, but ready to test. Contains HADOOP-18028. High performance S3A input stream (apache#4109) Contributed by Bhalchandra Pandit. HADOOP-18180. Replace use of twitter util-core with java futures (apache#4115) Contributed by PJ Fanning. HADOOP-18177. Document prefetching architecture. (apache#4205) Contributed by Ahmar Suhail HADOOP-18175. fix test failures with prefetching s3a input stream (apache#4212) Contributed by Monthon Klongklaew HADOOP-18231. S3A prefetching: fix failing tests & drain stream async. (apache#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail HADOOP-18254. Disable S3A prefetching by default. (apache#4469) Contributed by Ahmar Suhail HADOOP-18190. Collect IOStatistics during S3A prefetching (apache#4458) This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail HADOOP-18379 rebase feature/HADOOP-18028-s3a-prefetch to trunk HADOOP-18187. Convert s3a prefetching to use JavaDoc for fields and enums. HADOOP-18318. Update class names to be clear they belong to S3A prefetching Contributed by Steve Loughran Change-Id: I3eca19564dc0c0cb83184f4a42605dbafd908937

This is the the preview release of the HADOOP-18028 S3A performance input stream. It is still stabilizing, but ready to test. Contains HADOOP-18028. High performance S3A input stream (#4109) Contributed by Bhalchandra Pandit. HADOOP-18180. Replace use of twitter util-core with java futures (#4115) Contributed by PJ Fanning. HADOOP-18177. Document prefetching architecture. (#4205) Contributed by Ahmar Suhail HADOOP-18175. fix test failures with prefetching s3a input stream (#4212) Contributed by Monthon Klongklaew HADOOP-18231. S3A prefetching: fix failing tests & drain stream async. (#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail HADOOP-18254. Disable S3A prefetching by default. (#4469) Contributed by Ahmar Suhail HADOOP-18190. Collect IOStatistics during S3A prefetching (#4458) This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail HADOOP-18379 rebase feature/HADOOP-18028-s3a-prefetch to trunk HADOOP-18187. Convert s3a prefetching to use JavaDoc for fields and enums. HADOOP-18318. Update class names to be clear they belong to S3A prefetching Contributed by Steve Loughran

This is the the preview release of the HADOOP-18028 S3A performance input stream. It is still stabilizing, but ready to test. Contains HADOOP-18028. High performance S3A input stream (apache#4109) Contributed by Bhalchandra Pandit. HADOOP-18180. Replace use of twitter util-core with java futures (apache#4115) Contributed by PJ Fanning. HADOOP-18177. Document prefetching architecture. (apache#4205) Contributed by Ahmar Suhail HADOOP-18175. fix test failures with prefetching s3a input stream (apache#4212) Contributed by Monthon Klongklaew HADOOP-18231. S3A prefetching: fix failing tests & drain stream async. (apache#4386) * adds in new test for prefetching input stream * creates streamStats before opening stream * updates numBlocks calculation method * fixes ITestS3AOpenCost.testOpenFileLongerLength * drains stream async * fixes failing unit test Contributed by Ahmar Suhail HADOOP-18254. Disable S3A prefetching by default. (apache#4469) Contributed by Ahmar Suhail HADOOP-18190. Collect IOStatistics during S3A prefetching (apache#4458) This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with new statistic names in StreamStatistics. This stream is not (yet) IOStatisticsContext aware. Contributed by Ahmar Suhail HADOOP-18379 rebase feature/HADOOP-18028-s3a-prefetch to trunk HADOOP-18187. Convert s3a prefetching to use JavaDoc for fields and enums. HADOOP-18318. Update class names to be clear they belong to S3A prefetching Contributed by Steve Loughran

ahmarsuhail added 9 commits June 15, 2022 14:33

adds in iostats for prefetching ops count

0224784

adds in gauge for file cache count

d909226

decrement file cache gauge when deleting

6e037fc

adds in duration stats

e3684ec

rename prefetch stats method

ae352bb

adds in active memory stats

0943890

update memory in use on close()

8d9b327

cleanup and formatting changes

bb40655

fixes checkstyle errors

c121694

ahmarsuhail commented Jun 20, 2022

View reviewed changes

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java Outdated Show resolved Hide resolved

ahmarsuhail changed the title ~~Hadoop-18190. Adds iostats for prefetching~~ HADOOP-18190. Adds iostats for prefetching Jun 20, 2022

monthonk suggested changes Jun 21, 2022

View reviewed changes

changes as per code review

9f150cb

monthonk approved these changes Jun 24, 2022

View reviewed changes

apache deleted a comment from hadoop-yetus Jul 15, 2022

steveloughran requested changes Jul 15, 2022

View reviewed changes

Merge branch 'feature-HADOOP-18028-s3a-prefetch' into HADOOP-18190-pr…

65f5ee7

…efetching-add-iostats

move prefetching stat names to StreamStatistics

becdd66

steveloughran requested changes Jul 18, 2022

View reviewed changes

ahmarsuhail added 2 commits July 20, 2022 11:24

adds in stat for failed prefetched ops, assertions on stats in ITestS…

8a3f38d

…3PrefetchingInputStream

fixes spacing

1d710f1

steveloughran requested changes Jul 20, 2022

View reviewed changes

ahmarsuhail added 2 commits July 21, 2022 15:47

addresses review comments, adds in duration tracker for prefetch ops

66fb591

uses newInputStreamStatisitics()

62d1b8d

ahmarsuhail added 2 commits July 22, 2022 09:29

fixes formatting errors

5834ea9

adds in null checks for tracker

e556425

fixes indentation

492b2ef

steveloughran reviewed Jul 25, 2022

View reviewed changes

fixes package def indentation

021c98a

steveloughran merged commit a964fbf into apache:feature-HADOOP-18028-s3a-prefetch Jul 26, 2022

steveloughran mentioned this pull request Aug 17, 2022

HADOOP-18028. High performance S3A input stream #4752

Merged

4 tasks

steveloughran mentioned this pull request Apr 14, 2023

HADOOP-18028. High performance S3A input stream (#4752) #5559

Closed

4 tasks


		import org.apache.hadoop.fs.statistics.IOStatisticsSource;

		public interface PrefetchingStatistics extends IOStatisticsSource {

HADOOP-18190. Adds iostats for prefetching #4458

HADOOP-18190. Adds iostats for prefetching #4458

Uh oh!

Conversation

ahmarsuhail commented Jun 18, 2022

Description of PR

How was this patch tested?

Uh oh!

Uh oh!

monthonk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

monthonk left a comment

Choose a reason for hiding this comment

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

testing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Jul 18, 2022

Uh oh!

hadoop-yetus commented Jul 18, 2022

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahmarsuhail commented Jul 19, 2022

Uh oh!

hadoop-yetus commented Jul 20, 2022

Uh oh!

hadoop-yetus commented Jul 20, 2022

Uh oh!

steveloughran commented Jul 20, 2022

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Jul 21, 2022

Uh oh!

hadoop-yetus commented Jul 21, 2022

Uh oh!

hadoop-yetus commented Jul 22, 2022

Uh oh!