HADOOP-16379: S3AInputStream#unbuffer should merge input stream stats into fs-wide stats #983

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Closed

sahilTakiar wants to merge 1 commit into apache:trunk from sahilTakiar:HADOOP-16379

Contributor

sahilTakiar commented Jun 18, 2019

HADOOP-16379: S3AInputStream#unbuffer should merge input stream stats into fs-wide stats

Adds a new method to InputStreamStatistics called merge which allows users to periodically merge the stats into the fs-wide stats
Added new unit tests to validate the calling unbuffer merges the stream stats into the fs-wide stats

Testing:

Ran S3A tests against US East (N. Virginia)

hadoop-yetus commented Jun 18, 2019

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	86	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1155	trunk passed
+1	compile	33	trunk passed
+1	checkstyle	24	trunk passed
+1	mvnsite	40	trunk passed
+1	shadedclient	777	branch has no errors when building and testing our client artifacts.
+1	javadoc	25	trunk passed
0	spotbugs	65	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	62	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	33	the patch passed
+1	compile	31	the patch passed
+1	javac	31	the patch passed
-0	checkstyle	19	hadoop-tools/hadoop-aws: The patch generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19)
+1	mvnsite	36	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	819	patch has no errors when building and testing our client artifacts.
+1	javadoc	24	the patch passed
+1	findbugs	67	the patch passed
		_ Other Tests _
+1	unit	282	hadoop-aws in the patch passed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		3616

Subsystem	Report/Notes
Docker	Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-983/1/artifact/out/Dockerfile
GITHUB PR	#983
JIRA Issue	HADOOP-16379
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux f1310bc0ac26 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 62ad988
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-983/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-983/1/testReport/
Max. process+thread count	340 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-983/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

steveloughran reviewed

View reviewed changes

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AUnbuffer.java Outdated Show resolved Hide resolved

steveloughran reviewed

View reviewed changes

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AUnbuffer.java Outdated

Contributor

steveloughran Jun 18, 2019

use org.apache.hadoop.fs.s3a.S3ATestUtils.MetricDiff for measuring and asserting on state of metrics, as it generates meaningful error messages on assert failures.

steveloughran reviewed

View reviewed changes

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AUnbuffer.java Outdated Show resolved Hide resolved

steveloughran reviewed

View reviewed changes

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AUnbuffer.java Outdated

Contributor

steveloughran Jun 18, 2019

ContractTestUtils.writeDataset

steveloughran reviewed

View reviewed changes

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AUnbuffer.java Outdated

Contributor

steveloughran Jun 18, 2019 •

edited

Loading

needs to be deleted in a finally clause

Also, I don't think it is needed at all. MetricDiff is designed to do asserts over changes in statistics.

steveloughran reviewed

View reviewed changes

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AUnbuffer.java Outdated

Contributor

steveloughran Jun 18, 2019

superclass deleteTestDirInTeardown() should do this. If it doesn't, guard the call with checks for filesystem and destpath not being null, so it won't hide the stack traces of a failure in setup

Contributor

steveloughran commented Jun 18, 2019

LGTM, some changes around the testing. Checkstyle has a complaint which needs to be addressed


          HADOOP-16379: S3AInputStream#unbuffer should merge input stream stats…

1928b33

… into fs-wide stats

sahilTakiar force-pushed the HADOOP-16379 branch from 6ef1aed to 1928b33 Compare

June 18, 2019 17:44

hadoop-yetus commented Jun 18, 2019

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	71	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1116	trunk passed
+1	compile	34	trunk passed
+1	checkstyle	22	trunk passed
+1	mvnsite	37	trunk passed
+1	shadedclient	766	branch has no errors when building and testing our client artifacts.
+1	javadoc	24	trunk passed
0	spotbugs	57	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	55	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	30	the patch passed
+1	compile	27	the patch passed
+1	javac	27	the patch passed
+1	checkstyle	17	the patch passed
+1	mvnsite	32	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	802	patch has no errors when building and testing our client artifacts.
+1	javadoc	22	the patch passed
+1	findbugs	70	the patch passed
		_ Other Tests _
+1	unit	283	hadoop-aws in the patch passed.
+1	asflicense	28	The patch does not generate ASF License warnings.
		3530

Subsystem	Report/Notes
Docker	Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-983/2/artifact/out/Dockerfile
GITHUB PR	#983
JIRA Issue	HADOOP-16379
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 4f12c5432e99 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `3c1a1ce`
Default Java	1.8.0_212
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-983/2/testReport/
Max. process+thread count	320 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-983/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Contributor Author

sahilTakiar commented Jun 18, 2019

@steveloughran addressed your comments. Hadoop QA looks happy as well.

Re-ran the S3A tests.

Contributor

steveloughran commented Jun 20, 2019

LGTM: +1

Contributor

steveloughran commented Jun 20, 2019

committed to trunk. thanks!

steveloughran closed this

shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request


          Fixing the testSinkEndToEndWithKeyWithNullRecords test case (apache#983)

4afeb51

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet