Skip to content

Conversation

@steveloughran
Copy link
Contributor

This patch adds to hadoop-common the IOStatistics API
to query IO object instances (filesystems, streams, remote iterators)
for statistics on their I/O Usage: operation count and min/max/mean
durations.

Contributed by Steve Loughran.

This patch adds to hadoop-common the IOStatistics API
to query IO object instances (filesystems, streams, remote iterators)
for statistics on their I/O Usage: operation count and min/max/mean
durations.

Contributed by Steve Loughran.

Change-Id: I910cd4679abf4fffefed5b71f77d24097a04c6ae
@steveloughran
Copy link
Contributor Author

This is the hadoop-common part of #2553, which @bgaborg has already approved. I'm just pushing this bit of the patch through yetus before the final merge, and will follow with the MR and hadoop-aws bits.

@mehakmeet : once this is in you'll be able to sync your abfs code up

@hadoop-yetus

This comment has been minimized.

@steveloughran
Copy link
Contributor Author

All deprecation warnings will go away. Style and javadocs will be corrected

./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:41:   * @deprecated use {@link org.apache.hadoop.util.functional.FunctionRaisingIOE}: Line is longer than 80 characters (found 81). [LineLength]
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:54:   * @deprecated use {@link org.apache.hadoop.util.functional.BiFunctionRaisingIOE}: Line is longer than 80 characters (found 83). [LineLength]
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FunctionsRaisingIOE.java:65:   * @deprecated use {@link org.apache.hadoop.util.functional.CallableRaisingIOE}: Line is longer than 80 characters (found 81). [LineLength]
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/impl/IOStatisticsBinding.java:450:      DurationTracker tracker = createTracker(factory, statistic);: 'method def' child has incorrect indentation level 6, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/impl/IOStatisticsBinding.java:451:      try {: 'try' has incorrect indentation level 6, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/impl/IOStatisticsBinding.java:453:        input.apply();: 'try' child has incorrect indentation level 8, expected level should be 6. [Indentation]
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/impl/IOStatisticsBinding.java:454:      } catch (IOException | RuntimeException e) {: 'try rcurly' has incorrect indentation level 6, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/impl/IOStatisticsBinding.java:456:        tracker.failed();: 'catch' child has incorrect indentation level 8, expected level should be 6. [Indentation]
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/impl/IOStatisticsBinding.java:458:        throw e;: 'catch' child has incorrect indentation level 8, expected level should be 6. [Indentation]
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/impl/IOStatisticsBinding.java:459:      } finally {: 'catch rcurly' has incorrect indentation level 6, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/impl/IOStatisticsBinding.java:463:        tracker.close();: 'finally' child has incorrect indentation level 8, expected level should be 6. [Indentation]
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/impl/IOStatisticsBinding.java:464:      }: 'finally rcurly' has incorrect indentation level 6, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/package-info.java:28: * {@link org.apache.hadoop.fs.statistics.IOStatisticsSource#getIOStatistics()} .: Line is longer than 80 characters (found 81). [LineLength]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractStreamIOStatisticsTest.java:77:  /**: First sentence should end with a period. [JavadocStyle]

Change-Id: I0f76cf9f451af64eefd5193a03ae0f03084c1065
@steveloughran steveloughran changed the title HADOOP-16830. Add Public IOStatistics API. HADOOP-17450. Add Public IOStatistics API. Dec 30, 2020
@hadoop-yetus

This comment has been minimized.

Change-Id: I03dabadccec7f02aa399166ceb1fd2757651a72a
@steveloughran
Copy link
Contributor Author

lots of javadoc complaints about <p></p> constructs; I'd been using them because the rendered javadocs weren't as well split up as the unformatted ones in the text. But if javadoc complains that much -blank lines it is.

@hadoop-yetus

This comment has been minimized.

@steveloughran
Copy link
Contributor Author

This is an out of memory error. as I only changed javadocs since the last successful build, I'm going to do the build and javadoc audit locally, and if good: finish off that merge into trunk only.

Then I'll followup with the later changes, again, trunk, until I'm happy to do the CP into branch-3.3

@steveloughran steveloughran merged commit 99d08a1 into apache:trunk Dec 31, 2020
steveloughran added a commit to steveloughran/hadoop that referenced this pull request Jan 12, 2021
This is the API and implementation classes of HADOOP-16830,
which allows callers to query IO object instances
(filesystems, streams, remote iterators, ...) and other classes
for statistics on their I/O Usage: operation count and min/max/mean
durations.

New Packages

org.apache.hadoop.fs.statistics.
  Public API, including:
    IOStatisticsSource
    IOStatistics
    IOStatisticsSnapshot (seralizable to java objects and json)
    +helper classes for logging and integration
    BufferedIOStatisticsInputStream
       implements IOStatisticsSource and StreamCapabilities
     BufferedIOStatisticsOutputStream
       implements IOStatisticsSource, Syncable and StreamCapabilities

org.apache.hadoop.fs.statistics.impl
  Implementation classes for internal use.

org.apache.hadoop.util.functional
  functional programming support for RemoteIterators and
  other operations which raise IOEs; all wrapper classes
  implement and propagate IOStatisticsSource

Contributed by Steve Loughran.

Change-Id: If56e8db2981613ff689c39239135e44feb25f78e
asfgit pushed a commit that referenced this pull request Jan 14, 2021
This is the API and implementation classes of HADOOP-16830,
which allows callers to query IO object instances
(filesystems, streams, remote iterators, ...) and other classes
for statistics on their I/O Usage: operation count and min/max/mean
durations.

New Packages

org.apache.hadoop.fs.statistics.
  Public API, including:
    IOStatisticsSource
    IOStatistics
    IOStatisticsSnapshot (seralizable to java objects and json)
    +helper classes for logging and integration
    BufferedIOStatisticsInputStream
       implements IOStatisticsSource and StreamCapabilities
     BufferedIOStatisticsOutputStream
       implements IOStatisticsSource, Syncable and StreamCapabilities

org.apache.hadoop.fs.statistics.impl
  Implementation classes for internal use.

org.apache.hadoop.util.functional
  functional programming support for RemoteIterators and
  other operations which raise IOEs; all wrapper classes
  implement and propagate IOStatisticsSource

Contributed by Steve Loughran.

Change-Id: If56e8db2981613ff689c39239135e44feb25f78e
@steveloughran steveloughran deleted the s3/HADOOP-16830-iostatistics-commit-common branch October 15, 2021 19:48
jojochuang pushed a commit to jojochuang/hadoop that referenced this pull request May 23, 2023
This is the API and implementation classes of HADOOP-16830,
which allows callers to query IO object instances
(filesystems, streams, remote iterators, ...) and other classes
for statistics on their I/O Usage: operation count and min/max/mean
durations.

New Packages

org.apache.hadoop.fs.statistics.
  Public API, including:
    IOStatisticsSource
    IOStatistics
    IOStatisticsSnapshot (seralizable to java objects and json)
    +helper classes for logging and integration
    BufferedIOStatisticsInputStream
       implements IOStatisticsSource and StreamCapabilities
     BufferedIOStatisticsOutputStream
       implements IOStatisticsSource, Syncable and StreamCapabilities

org.apache.hadoop.fs.statistics.impl
  Implementation classes for internal use.

org.apache.hadoop.util.functional
  functional programming support for RemoteIterators and
  other operations which raise IOEs; all wrapper classes
  implement and propagate IOStatisticsSource

Contributed by Steve Loughran.

Change-Id: If56e8db2981613ff689c39239135e44feb25f78e
deepakdamri pushed a commit to acceldata-io/hadoop that referenced this pull request Jan 21, 2025
This is the API and implementation classes of HADOOP-16830,
which allows callers to query IO object instances
(filesystems, streams, remote iterators, ...) and other classes
for statistics on their I/O Usage: operation count and min/max/mean
durations.

New Packages

org.apache.hadoop.fs.statistics.
  Public API, including:
    IOStatisticsSource
    IOStatistics
    IOStatisticsSnapshot (seralizable to java objects and json)
    +helper classes for logging and integration
    BufferedIOStatisticsInputStream
       implements IOStatisticsSource and StreamCapabilities
     BufferedIOStatisticsOutputStream
       implements IOStatisticsSource, Syncable and StreamCapabilities

org.apache.hadoop.fs.statistics.impl
  Implementation classes for internal use.

org.apache.hadoop.util.functional
  functional programming support for RemoteIterators and
  other operations which raise IOEs; all wrapper classes
  implement and propagate IOStatisticsSource

Contributed by Steve Loughran.

Change-Id: If56e8db2981613ff689c39239135e44feb25f78e
deepakdamri pushed a commit to acceldata-io/hadoop that referenced this pull request Jan 21, 2025
This is the API and implementation classes of HADOOP-16830,
which allows callers to query IO object instances
(filesystems, streams, remote iterators, ...) and other classes
for statistics on their I/O Usage: operation count and min/max/mean
durations.

New Packages

org.apache.hadoop.fs.statistics.
  Public API, including:
    IOStatisticsSource
    IOStatistics
    IOStatisticsSnapshot (seralizable to java objects and json)
    +helper classes for logging and integration
    BufferedIOStatisticsInputStream
       implements IOStatisticsSource and StreamCapabilities
     BufferedIOStatisticsOutputStream
       implements IOStatisticsSource, Syncable and StreamCapabilities

org.apache.hadoop.fs.statistics.impl
  Implementation classes for internal use.

org.apache.hadoop.util.functional
  functional programming support for RemoteIterators and
  other operations which raise IOEs; all wrapper classes
  implement and propagate IOStatisticsSource

Contributed by Steve Loughran.

Change-Id: If56e8db2981613ff689c39239135e44feb25f78e
deepakdamri pushed a commit to acceldata-io/hadoop that referenced this pull request Jan 21, 2025
This is the API and implementation classes of HADOOP-16830,
which allows callers to query IO object instances
(filesystems, streams, remote iterators, ...) and other classes
for statistics on their I/O Usage: operation count and min/max/mean
durations.

New Packages

org.apache.hadoop.fs.statistics.
  Public API, including:
    IOStatisticsSource
    IOStatistics
    IOStatisticsSnapshot (seralizable to java objects and json)
    +helper classes for logging and integration
    BufferedIOStatisticsInputStream
       implements IOStatisticsSource and StreamCapabilities
     BufferedIOStatisticsOutputStream
       implements IOStatisticsSource, Syncable and StreamCapabilities

org.apache.hadoop.fs.statistics.impl
  Implementation classes for internal use.

org.apache.hadoop.util.functional
  functional programming support for RemoteIterators and
  other operations which raise IOEs; all wrapper classes
  implement and propagate IOStatisticsSource

Contributed by Steve Loughran.

Change-Id: If56e8db2981613ff689c39239135e44feb25f78e
deepakdamri pushed a commit to acceldata-io/hadoop that referenced this pull request Jan 21, 2025
This is the API and implementation classes of HADOOP-16830,
which allows callers to query IO object instances
(filesystems, streams, remote iterators, ...) and other classes
for statistics on their I/O Usage: operation count and min/max/mean
durations.

New Packages

org.apache.hadoop.fs.statistics.
  Public API, including:
    IOStatisticsSource
    IOStatistics
    IOStatisticsSnapshot (seralizable to java objects and json)
    +helper classes for logging and integration
    BufferedIOStatisticsInputStream
       implements IOStatisticsSource and StreamCapabilities
     BufferedIOStatisticsOutputStream
       implements IOStatisticsSource, Syncable and StreamCapabilities

org.apache.hadoop.fs.statistics.impl
  Implementation classes for internal use.

org.apache.hadoop.util.functional
  functional programming support for RemoteIterators and
  other operations which raise IOEs; all wrapper classes
  implement and propagate IOStatisticsSource

Contributed by Steve Loughran.

Change-Id: If56e8db2981613ff689c39239135e44feb25f78e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants