HADOOP-17016. Adding Common Counters in ABFS #1991

mehakmeet · 2020-04-30T16:20:06Z

Common Counters to be Added in ABFS.

test run: mvn -T 1C -Dparallel-tests=abfs clean verify
Region: East US, West US

UT:

[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.016 s - in org.apache.hadoop.fs.azurebfs.TestAbfsStatistics

IT:

[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsStatistics
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.642 s - in org.apache.hadoop.fs.azurebfs.ITestAbfsStatistics

hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsStatistics.java

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java

hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsStatistics.java

hadoop-yetus · 2020-04-30T18:05:41Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	22m 26s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	19m 37s	trunk passed
+1 💚	compile	0m 33s	trunk passed
+1 💚	checkstyle	0m 26s	trunk passed
+1 💚	mvnsite	0m 35s	trunk passed
+1 💚	shadedclient	14m 50s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 27s	trunk passed
+0 🆗	spotbugs	0m 52s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 50s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 28s	the patch passed
+1 💚	compile	0m 24s	the patch passed
+1 💚	javac	0m 24s	the patch passed
-0 ⚠️	checkstyle	0m 16s	hadoop-tools/hadoop-azure: The patch generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1)
+1 💚	mvnsite	0m 27s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	13m 55s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 24s	the patch passed
+1 💚	findbugs	0m 55s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 21s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 31s	The patch does not generate ASF License warnings.
		79m 57s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1991/1/artifact/out/Dockerfile
GITHUB PR	#1991
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 70da264622db 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `6bdab37`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1991/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1991/1/testReport/
Max. process+thread count	414 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1991/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-04-30T18:49:30Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 23s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	22m 6s	trunk passed
+1 💚	compile	0m 30s	trunk passed
+1 💚	checkstyle	0m 24s	trunk passed
+1 💚	mvnsite	0m 33s	trunk passed
+1 💚	shadedclient	16m 22s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 26s	trunk passed
+0 🆗	spotbugs	0m 51s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 50s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 27s	the patch passed
+1 💚	compile	0m 23s	the patch passed
+1 💚	javac	0m 23s	the patch passed
-0 ⚠️	checkstyle	0m 14s	hadoop-tools/hadoop-azure: The patch generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1)
+1 💚	mvnsite	0m 25s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	15m 28s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 20s	the patch passed
+1 💚	findbugs	0m 55s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 17s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 27s	The patch does not generate ASF License warnings.
		63m 54s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1991/2/artifact/out/Dockerfile
GITHUB PR	#1991
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux b695a7c7a2d1 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `6bdab37`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1991/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1991/2/testReport/
Max. process+thread count	343 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1991/2/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsInstrumentation.java

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java

hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAbfsStatistics.java

hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsStatistics.java

mukund-thakur

Production code looks fine. Please fix the comments.
@steveloughran Could you also take a look once? Thanks.

hadoop-yetus · 2020-05-12T12:54:31Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 3s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	23m 11s	trunk passed
+1 💚	compile	0m 32s	trunk passed
+1 💚	checkstyle	0m 23s	trunk passed
+1 💚	mvnsite	0m 34s	trunk passed
+1 💚	shadedclient	16m 17s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 24s	trunk passed
+0 🆗	spotbugs	0m 58s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 55s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 32s	the patch passed
+1 💚	compile	0m 26s	the patch passed
+1 💚	javac	0m 26s	the patch passed
+1 💚	checkstyle	0m 16s	the patch passed
+1 💚	mvnsite	0m 28s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	16m 6s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 23s	the patch passed
+1 💚	findbugs	1m 3s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 27s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 33s	The patch does not generate ASF License warnings.
		66m 12s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1991/3/artifact/out/Dockerfile
GITHUB PR	#1991
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux c7902cc0ed5f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `0fe4903`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1991/3/testReport/
Max. process+thread count	414 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1991/3/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsStatistics.java

hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAbfsStatistics.java

mukund-thakur

Some minor changes pending. Rest looks fine.

hadoop-yetus · 2020-05-19T12:51:29Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 26s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	21m 6s	trunk passed
+1 💚	compile	0m 27s	trunk passed
+1 💚	checkstyle	0m 19s	trunk passed
+1 💚	mvnsite	0m 30s	trunk passed
+1 💚	shadedclient	16m 1s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 22s	trunk passed
+0 🆗	spotbugs	0m 51s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 50s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 27s	the patch passed
+1 💚	compile	0m 22s	the patch passed
+1 💚	javac	0m 23s	the patch passed
+1 💚	checkstyle	0m 14s	the patch passed
+1 💚	mvnsite	0m 25s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	14m 53s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 19s	the patch passed
+1 💚	findbugs	0m 53s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 10s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 27s	The patch does not generate ASF License warnings.
		60m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1991/5/artifact/out/Dockerfile
GITHUB PR	#1991
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 6a2e44dfe836 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `d4e3640`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1991/5/testReport/
Max. process+thread count	345 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1991/5/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2020-05-20T12:15:33Z

can be done in one line ??

we're ok with imports of constants; its the package/method imports that are most brittle. Just use .*

steveloughran

LTGM

Key proposed changes

just use .* for the constant imports
move to interface + implementation of the metrics class
proposed test cleanup by adding custom assertion and so eliminate duplicate code.

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsInstrumentation.java

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java

hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsStatistics.java

hadoop-yetus · 2020-05-25T08:23:02Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	20m 56s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 3 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	21m 47s	trunk passed
+1 💚	compile	0m 27s	trunk passed
+1 💚	checkstyle	0m 21s	trunk passed
+1 💚	mvnsite	0m 30s	trunk passed
+1 💚	shadedclient	16m 8s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 22s	trunk passed
+0 🆗	spotbugs	0m 52s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 51s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 26s	the patch passed
+1 💚	compile	0m 21s	the patch passed
+1 💚	javac	0m 21s	the patch passed
-0 ⚠️	checkstyle	0m 14s	hadoop-tools/hadoop-azure: The patch generated 2 new + 3 unchanged - 0 fixed = 5 total (was 3)
+1 💚	mvnsite	0m 24s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	15m 13s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 19s	the patch passed
+1 💚	findbugs	0m 51s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 9s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 28s	The patch does not generate ASF License warnings.
		82m 18s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1991/6/artifact/out/Dockerfile
GITHUB PR	#1991
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux c1692b544083 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `f43a152`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1991/6/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1991/6/testReport/
Max. process+thread count	308 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1991/6/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2020-06-02T17:24:34Z

hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsStatistics.java

    fs.open(createFilePath);
    fs.append(createFilePath);
-    fs.rename(createFilePath, destCreateFilePath);
+    assertTrue(fs.rename(createFilePath, destCreateFilePath));


nit: There's a ContractTestUtils rename which is a bit more informative on rename failure, but I Don't see this failing enough to worry about it.

I'll make sure to use that going forward.

steveloughran · 2020-06-02T17:28:59Z

Ok, all is good. I'd point you at ContractTestUtils for future rename/exists checks in tests, as it has better error reporting.

+1

Contributed by: Mehakmeet Singh. Change-Id: Ib84e7a42f28e064df4c6204fcce33e573360bf42

steveloughran · 2020-06-02T17:33:00Z

merged to trunk

Contributed by: Mehakmeet Singh. Change-Id: Ib84e7a42f28e064df4c6204fcce33e573360bf42

Contributed by: Mehakmeet Singh. Change-Id: Ib84e7a42f28e064df4c6204fcce33e573360bf42 (cherry picked from commit 7f486f0)

HADOOP-17016. Adding Common Counters in ABFS

b515330

mehakmeet commented Apr 30, 2020

View reviewed changes

hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsStatistics.java Outdated Show resolved Hide resolved

mehakmeet commented Apr 30, 2020

View reviewed changes

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java Outdated Show resolved Hide resolved

mehakmeet commented Apr 30, 2020

View reviewed changes

hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsStatistics.java Outdated Show resolved Hide resolved

HADOOP-17016. Self-review comments fix

41d08b1

mukund-thakur reviewed May 12, 2020

View reviewed changes

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsInstrumentation.java Outdated Show resolved Hide resolved

mukund-thakur reviewed May 12, 2020

View reviewed changes

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java Show resolved Hide resolved

mukund-thakur reviewed May 12, 2020

View reviewed changes

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java Outdated Show resolved Hide resolved

mukund-thakur reviewed May 12, 2020

View reviewed changes

hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAbfsStatistics.java Outdated Show resolved Hide resolved

mukund-thakur reviewed May 12, 2020

View reviewed changes

hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAbfsStatistics.java Outdated Show resolved Hide resolved

mukund-thakur reviewed May 12, 2020

View reviewed changes

hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAbfsStatistics.java Show resolved Hide resolved