HDFS-14856. Fetch file ACLs while mounting external store. #1478

ashvina · 2019-09-19T17:19:08Z

Addresses https://issues.apache.org/jira/browse/HDFS-14856.
If configuration dfs.namenode.mount.acls.enabled is set (true), FsTreeWalk will fetch ACLs of files on the external storage system provided at the time of mount.

goiri · 2019-09-19T18:04:03Z

hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java

Should we refer to PROVIDED here?

Agree. WDYT about dfs.namenode.provided.acls.enabled?
Keep namenode since the config is a NN config.
Replaced mount with provided so that the config name is consistent.

Better than before for sure.
Not sure about the end policy for distinguishing namenode from not.
It should be consistent.

goiri · 2019-09-19T18:05:09Z

hadoop-tools/hadoop-fs2img/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSTreeWalk.java

Extract for readability.

goiri · 2019-09-19T18:05:24Z

hadoop-tools/hadoop-fs2img/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSTreeWalk.java

Do we support adding null to the acls?
Is there a finer grain exception then IOException?

ACL can be null.
IOException comes from the FileSystems API, AclStatus getAclStatus(Path path) throws IOException

goiri · 2019-09-19T18:06:21Z

hadoop-tools/hadoop-fs2img/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSTreeWalk.java

This one we don't care about the IOE?

Good catch. The scanner will not fail if the remote FS does not support ACLs. All other IOException fail the operation in all paths now. Thanks !

@ashvina - is it possible to handle the exception the same way in both the constructors?

goiri · 2019-09-19T18:06:40Z

hadoop-tools/hadoop-fs2img/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSTreeWalk.java

goiri · 2019-09-19T18:08:03Z

hadoop-tools/hadoop-fs2img/src/main/java/org/apache/hadoop/hdfs/server/namenode/TreePath.java

goiri · 2019-09-19T18:08:18Z

hadoop-tools/hadoop-fs2img/src/main/java/org/apache/hadoop/hdfs/server/namenode/TreeWalk.java

goiri · 2019-09-19T18:09:25Z

...op-tools/hadoop-fs2img/src/main/java/org/apache/hadoop/hdfs/server/namenode/UGIResolver.java

getUser()
Is this static?

Why do we have this just for doing a get?

The user mapping behavior can be different in a subclass. For e.g. In this case SingleUGIResolver maps all users to a single user.

goiri · 2019-09-19T18:09:43Z

...op-tools/hadoop-fs2img/src/main/java/org/apache/hadoop/hdfs/server/namenode/UGIResolver.java

static getACLEntries()

goiri · 2019-09-19T18:10:40Z

...op-tools/hadoop-fs2img/src/main/java/org/apache/hadoop/hdfs/server/namenode/UGIResolver.java

Extract getName()

hadoop-yetus · 2019-09-19T23:13:02Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	136	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	83	Maven dependency ordering for branch
+1	mvninstall	1561	trunk passed
+1	compile	1433	trunk passed
+1	checkstyle	206	trunk passed
+1	mvnsite	142	trunk passed
+1	shadedclient	1201	branch has no errors when building and testing our client artifacts.
+1	javadoc	94	trunk passed
0	spotbugs	46	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	239	trunk passed
		_ Patch Compile Tests _
0	mvndep	24	Maven dependency ordering for patch
+1	mvninstall	89	the patch passed
+1	compile	1142	the patch passed
+1	javac	1142	the patch passed
-0	checkstyle	161	root: The patch generated 4 new + 452 unchanged - 1 fixed = 456 total (was 453)
+1	mvnsite	111	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	2	The patch has no ill-formed XML file.
+1	shadedclient	776	patch has no errors when building and testing our client artifacts.
+1	javadoc	95	the patch passed
+1	findbugs	248	the patch passed
		_ Other Tests _
+1	unit	6732	hadoop-hdfs in the patch passed.
+1	unit	37	hadoop-fs2img in the patch passed.
+1	asflicense	50	The patch does not generate ASF License warnings.
		14550

Subsystem	Report/Notes
Docker	Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/1/artifact/out/Dockerfile
GITHUB PR	#1478
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux f1e46b322601 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `126ef77`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/1/artifact/out/diff-checkstyle-root.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/1/testReport/
Max. process+thread count	2880 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-fs2img U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

virajith · 2019-09-20T16:57:44Z

hadoop-tools/hadoop-fs2img/src/main/java/org/apache/hadoop/hdfs/server/namenode/TreePath.java

So, this is not currently written to the image?

Yes. I am proposing to fix the scanner in the PR which is independent of the ImageWriter. The consumer of the scanner can choose to ignore the ACL.

Can you make this explicit (e.g,, add javadoc for this method)?

hadoop-yetus · 2019-09-21T02:29:09Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	33	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	59	Maven dependency ordering for branch
+1	mvninstall	1287	trunk passed
+1	compile	1158	trunk passed
+1	checkstyle	211	trunk passed
+1	mvnsite	114	trunk passed
-1	shadedclient	422	branch has errors when building and testing our client artifacts.
+1	javadoc	115	trunk passed
0	spotbugs	47	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	244	trunk passed
		_ Patch Compile Tests _
0	mvndep	24	Maven dependency ordering for patch
+1	mvninstall	92	the patch passed
+1	compile	1141	the patch passed
+1	javac	1141	the patch passed
-0	checkstyle	169	root: The patch generated 9 new + 452 unchanged - 1 fixed = 461 total (was 453)
+1	mvnsite	110	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	2	The patch has no ill-formed XML file.
-1	shadedclient	52	patch has errors when building and testing our client artifacts.
+1	javadoc	115	the patch passed
+1	findbugs	264	the patch passed
		_ Other Tests _
-1	unit	5827	hadoop-hdfs in the patch failed.
+1	unit	61	hadoop-fs2img in the patch passed.
+1	asflicense	55	The patch does not generate ASF License warnings.
		11538

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
	hadoop.hdfs.TestFileChecksumCompositeCrc

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/2/artifact/out/Dockerfile
GITHUB PR	#1478
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 0ffddb4b3ead 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `d7d6ec8`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/2/artifact/out/diff-checkstyle-root.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/2/testReport/
Max. process+thread count	2883 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-fs2img U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-09-21T02:31:11Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	35	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	24	Maven dependency ordering for branch
+1	mvninstall	1254	trunk passed
+1	compile	1153	trunk passed
+1	checkstyle	160	trunk passed
+1	mvnsite	114	trunk passed
-1	shadedclient	370	branch has errors when building and testing our client artifacts.
+1	javadoc	117	trunk passed
0	spotbugs	46	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	242	trunk passed
		_ Patch Compile Tests _
0	mvndep	24	Maven dependency ordering for patch
+1	mvninstall	102	the patch passed
+1	compile	1127	the patch passed
+1	javac	1127	the patch passed
-0	checkstyle	158	root: The patch generated 10 new + 452 unchanged - 1 fixed = 462 total (was 453)
+1	mvnsite	115	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	1	The patch has no ill-formed XML file.
-1	shadedclient	50	patch has errors when building and testing our client artifacts.
+1	javadoc	121	the patch passed
+1	findbugs	269	the patch passed
		_ Other Tests _
-1	unit	5828	hadoop-hdfs in the patch failed.
+1	unit	41	hadoop-fs2img in the patch passed.
+1	asflicense	50	The patch does not generate ASF License warnings.
		11403

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestDirectoryScanner
	hadoop.hdfs.TestFileChecksum
	hadoop.hdfs.TestLeaseRecovery2
	hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/3/artifact/out/Dockerfile
GITHUB PR	#1478
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 7f059105e6b8 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `d7d6ec8`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/3/artifact/out/diff-checkstyle-root.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/3/testReport/
Max. process+thread count	3376 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-fs2img U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

virajith · 2019-09-21T20:37:39Z

hadoop-tools/hadoop-fs2img/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSTreeWalk.java

@ashvina - is it possible to handle the exception the same way in both the constructors?

...op-tools/hadoop-fs2img/src/main/java/org/apache/hadoop/hdfs/server/namenode/UGIResolver.java

virajith · 2019-09-21T20:51:23Z

...tools/hadoop-fs2img/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSTreeWalk.java

change to javadoc style comment.

goiri · 2019-09-23T18:11:47Z

hadoop-tools/hadoop-fs2img/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSTreeWalk.java

Can we have a unit test checking for this string?

goiri · 2019-09-23T18:12:47Z

hadoop-tools/hadoop-fs2img/src/main/java/org/apache/hadoop/hdfs/server/namenode/TreePath.java

goiri · 2019-09-23T18:13:45Z

...op-tools/hadoop-fs2img/src/main/java/org/apache/hadoop/hdfs/server/namenode/UGIResolver.java

No. Static will not work as resolution methods in the subclass may not be static.

hadoop-yetus · 2019-09-24T01:25:18Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	135	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	91	Maven dependency ordering for branch
+1	mvninstall	1588	trunk passed
+1	compile	1497	trunk passed
+1	checkstyle	215	trunk passed
+1	mvnsite	145	trunk passed
+1	shadedclient	1342	branch has no errors when building and testing our client artifacts.
+1	javadoc	145	trunk passed
0	spotbugs	55	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	266	trunk passed
		_ Patch Compile Tests _
0	mvndep	29	Maven dependency ordering for patch
+1	mvninstall	117	the patch passed
+1	compile	1391	the patch passed
+1	javac	1391	the patch passed
-0	checkstyle	214	root: The patch generated 11 new + 452 unchanged - 1 fixed = 463 total (was 453)
+1	mvnsite	142	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	3	The patch has no ill-formed XML file.
+1	shadedclient	871	patch has no errors when building and testing our client artifacts.
+1	javadoc	144	the patch passed
+1	findbugs	286	the patch passed
		_ Other Tests _
-1	unit	6182	hadoop-hdfs in the patch failed.
+1	unit	62	hadoop-fs2img in the patch passed.
+1	asflicense	55	The patch does not generate ASF License warnings.
		14963

Reason	Tests
Failed junit tests	hadoop.cli.TestHDFSCLI
	hadoop.hdfs.TestDFSShell
	hadoop.hdfs.tools.TestDFSZKFailoverController

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/4/artifact/out/Dockerfile
GITHUB PR	#1478
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux b0dd932ec530 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `6cbe5d3`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/4/artifact/out/diff-checkstyle-root.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/4/testReport/
Max. process+thread count	3291 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-fs2img U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/4/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-09-24T01:43:28Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	46	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 3 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	68	Maven dependency ordering for branch
+1	mvninstall	1360	trunk passed
+1	compile	1172	trunk passed
+1	checkstyle	172	trunk passed
+1	mvnsite	130	trunk passed
+1	shadedclient	1077	branch has no errors when building and testing our client artifacts.
+1	javadoc	131	trunk passed
0	spotbugs	52	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	228	trunk passed
		_ Patch Compile Tests _
0	mvndep	27	Maven dependency ordering for patch
+1	mvninstall	91	the patch passed
+1	compile	957	the patch passed
+1	javac	957	the patch passed
-0	checkstyle	164	root: The patch generated 10 new + 452 unchanged - 1 fixed = 462 total (was 453)
+1	mvnsite	122	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	2	The patch has no ill-formed XML file.
+1	shadedclient	721	patch has no errors when building and testing our client artifacts.
+1	javadoc	149	the patch passed
+1	findbugs	247	the patch passed
		_ Other Tests _
-1	unit	5134	hadoop-hdfs in the patch failed.
+1	unit	54	hadoop-fs2img in the patch passed.
+1	asflicense	64	The patch does not generate ASF License warnings.
		12197

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDFSShell
	hadoop.hdfs.server.namenode.TestRedudantBlocks
	hadoop.cli.TestHDFSCLI
	hadoop.hdfs.TestReadStripedFileWithDNFailure
	hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy
	hadoop.hdfs.TestErasureCodingPolicies
	hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/5/artifact/out/Dockerfile
GITHUB PR	#1478
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 50395e82f257 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `6cbe5d3`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/5/artifact/out/diff-checkstyle-root.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/5/testReport/
Max. process+thread count	5262 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-fs2img U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/5/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-09-24T05:05:49Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	73	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 3 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	57	Maven dependency ordering for branch
+1	mvninstall	1184	trunk passed
+1	compile	1144	trunk passed
+1	checkstyle	195	trunk passed
+1	mvnsite	121	trunk passed
+1	shadedclient	1180	branch has no errors when building and testing our client artifacts.
+1	javadoc	116	trunk passed
0	spotbugs	44	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	227	trunk passed
		_ Patch Compile Tests _
0	mvndep	25	Maven dependency ordering for patch
+1	mvninstall	88	the patch passed
+1	compile	1089	the patch passed
+1	javac	1089	the patch passed
-0	checkstyle	194	root: The patch generated 9 new + 452 unchanged - 1 fixed = 461 total (was 453)
+1	mvnsite	121	the patch passed
+1	whitespace	1	The patch has no whitespace issues.
+1	xml	2	The patch has no ill-formed XML file.
+1	shadedclient	810	patch has no errors when building and testing our client artifacts.
+1	javadoc	112	the patch passed
+1	findbugs	242	the patch passed
		_ Other Tests _
-1	unit	6143	hadoop-hdfs in the patch failed.
+1	unit	42	hadoop-fs2img in the patch passed.
+1	asflicense	54	The patch does not generate ASF License warnings.
		13227

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDFSShell
	hadoop.cli.TestHDFSCLI
	hadoop.hdfs.TestDFSInputStream
	hadoop.hdfs.tools.TestDFSZKFailoverController

Subsystem	Report/Notes
Docker	Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/6/artifact/out/Dockerfile
GITHUB PR	#1478
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 1d8a733f6111 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / e8e7d7b
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/6/artifact/out/diff-checkstyle-root.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/6/testReport/
Max. process+thread count	2795 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-fs2img U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/6/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

virajith

+1 once the comments are resolved and yetus issues (mostly checkstyle; failed tests seem unrelated) are resolved.

hadoop-yetus · 2019-09-26T21:54:34Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	36	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 3 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	62	Maven dependency ordering for branch
+1	mvninstall	1256	trunk passed
+1	compile	1068	trunk passed
+1	checkstyle	177	trunk passed
+1	mvnsite	110	trunk passed
+1	shadedclient	1124	branch has no errors when building and testing our client artifacts.
+1	javadoc	113	trunk passed
0	spotbugs	43	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	224	trunk passed
		_ Patch Compile Tests _
0	mvndep	21	Maven dependency ordering for patch
+1	mvninstall	86	the patch passed
+1	compile	1010	the patch passed
+1	javac	1010	the patch passed
+1	checkstyle	173	root: The patch generated 0 new + 452 unchanged - 1 fixed = 452 total (was 453)
+1	mvnsite	109	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	1	The patch has no ill-formed XML file.
+1	shadedclient	795	patch has no errors when building and testing our client artifacts.
+1	javadoc	117	the patch passed
+1	findbugs	242	the patch passed
		_ Other Tests _
-1	unit	5220	hadoop-hdfs in the patch failed.
-1	unit	33	hadoop-fs2img in the patch failed.
-1	asflicense	53	The patch generated 2 ASF License warnings.
		12071

Reason	Tests
Failed junit tests	hadoop.hdfs.TestClientProtocolForPipelineRecovery
	hadoop.hdfs.server.balancer.TestBalancer
	hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy
	hadoop.hdfs.TestLeaseRecovery
	hadoop.hdfs.TestDFSStripedOutputStream
	hadoop.hdfs.tools.TestDFSZKFailoverController
	hadoop.hdfs.TestDecommissionWithStriped
	hadoop.hdfs.TestHDFSFileSystemContract
	hadoop.hdfs.TestWriteBlockGetsBlockLengthHint

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/7/artifact/out/Dockerfile
GITHUB PR	#1478
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 64825d0c72c7 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `06998a1`
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/7/artifact/out/patch-unit-hadoop-tools_hadoop-fs2img.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/7/testReport/
asflicense	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/7/artifact/out/patch-asflicense-problems.txt
Max. process+thread count	3669 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-fs2img U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/7/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-10-01T08:52:08Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	35	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 3 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	62	Maven dependency ordering for branch
+1	mvninstall	1193	trunk passed
+1	compile	1046	trunk passed
+1	checkstyle	174	trunk passed
+1	mvnsite	108	trunk passed
+1	shadedclient	1116	branch has no errors when building and testing our client artifacts.
+1	javadoc	111	trunk passed
0	spotbugs	42	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	225	trunk passed
		_ Patch Compile Tests _
0	mvndep	22	Maven dependency ordering for patch
+1	mvninstall	86	the patch passed
+1	compile	1008	the patch passed
+1	javac	1008	the patch passed
+1	checkstyle	174	root: The patch generated 0 new + 452 unchanged - 1 fixed = 452 total (was 453)
+1	mvnsite	112	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	2	The patch has no ill-formed XML file.
+1	shadedclient	792	patch has no errors when building and testing our client artifacts.
+1	javadoc	112	the patch passed
+1	findbugs	245	the patch passed
		_ Other Tests _
-1	unit	5122	hadoop-hdfs in the patch failed.
+1	unit	46	hadoop-fs2img in the patch passed.
+1	asflicense	51	The patch does not generate ASF License warnings.
		11865

Reason	Tests
Failed junit tests	hadoop.hdfs.tools.TestDFSZKFailoverController

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/8/artifact/out/Dockerfile
GITHUB PR	#1478
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux ed49fa9edae7 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 137546a
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/8/testReport/
Max. process+thread count	3412 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-fs2img U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/8/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

...tools/hadoop-fs2img/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSTreeWalk.java

virajith

@ashvina Sorry about the delay on this. I have a couple of minor comments. I am +1 on this once these are fixed.

...tools/hadoop-fs2img/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSTreeWalk.java

hadoop-yetus · 2019-10-11T20:10:40Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	91	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 3 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	28	Maven dependency ordering for branch
+1	mvninstall	1382	trunk passed
+1	compile	1100	trunk passed
+1	checkstyle	177	trunk passed
+1	mvnsite	112	trunk passed
+1	shadedclient	1136	branch has no errors when building and testing our client artifacts.
+1	javadoc	113	trunk passed
0	spotbugs	44	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	226	trunk passed
		_ Patch Compile Tests _
0	mvndep	22	Maven dependency ordering for patch
+1	mvninstall	87	the patch passed
+1	compile	1023	the patch passed
+1	javac	1023	the patch passed
-0	checkstyle	173	root: The patch generated 1 new + 453 unchanged - 1 fixed = 454 total (was 454)
+1	mvnsite	112	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	1	The patch has no ill-formed XML file.
+1	shadedclient	795	patch has no errors when building and testing our client artifacts.
+1	javadoc	112	the patch passed
+1	findbugs	238	the patch passed
		_ Other Tests _
-1	unit	5995	hadoop-hdfs in the patch failed.
+1	unit	43	hadoop-fs2img in the patch passed.
+1	asflicense	57	The patch does not generate ASF License warnings.
		13063

Reason	Tests
Failed junit tests	hadoop.hdfs.tools.TestDFSZKFailoverController
	hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots

Subsystem	Report/Notes
Docker	Client=19.03.3 Server=19.03.3 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/10/artifact/out/Dockerfile
GITHUB PR	#1478
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 78fab810aa8e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / ec86f42
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/10/artifact/out/diff-checkstyle-root.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/10/testReport/
Max. process+thread count	2680 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-fs2img U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/10/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-10-12T02:14:37Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	77	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 3 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	21	Maven dependency ordering for branch
+1	mvninstall	1270	trunk passed
+1	compile	1180	trunk passed
+1	checkstyle	177	trunk passed
+1	mvnsite	110	trunk passed
+1	shadedclient	1159	branch has no errors when building and testing our client artifacts.
+1	javadoc	111	trunk passed
0	spotbugs	44	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	222	trunk passed
		_ Patch Compile Tests _
0	mvndep	22	Maven dependency ordering for patch
+1	mvninstall	88	the patch passed
+1	compile	1027	the patch passed
+1	javac	1027	the patch passed
+1	checkstyle	217	root: The patch generated 0 new + 453 unchanged - 1 fixed = 453 total (was 454)
+1	mvnsite	116	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	1	The patch has no ill-formed XML file.
+1	shadedclient	808	patch has no errors when building and testing our client artifacts.
+1	javadoc	114	the patch passed
+1	findbugs	239	the patch passed
		_ Other Tests _
-1	unit	6094	hadoop-hdfs in the patch failed.
+1	unit	47	hadoop-fs2img in the patch passed.
+1	asflicense	54	The patch does not generate ASF License warnings.
		13185

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap
	hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots

Subsystem	Report/Notes
Docker	Client=19.03.3 Server=19.03.3 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/11/artifact/out/Dockerfile
GITHUB PR	#1478
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux c42458573b42 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / c561a70
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/11/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/11/testReport/
Max. process+thread count	2838 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-fs2img U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1478/11/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

virajith · 2019-10-14T16:44:14Z

Test failures are unrelated. Completing this PR.

goiri requested changes Sep 19, 2019

View reviewed changes

virajith reviewed Sep 20, 2019

View reviewed changes

virajith requested changes Sep 21, 2019

View reviewed changes

goiri reviewed Sep 23, 2019

View reviewed changes

hadoop-tools/hadoop-fs2img/src/main/java/org/apache/hadoop/hdfs/server/namenode/TreePath.java Outdated

Copy link

Member

goiri Sep 23, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Too long.

goiri reviewed Sep 23, 2019

View reviewed changes

virajith approved these changes Sep 24, 2019

View reviewed changes

ashvina force-pushed the fetchRemoteAcls branch from d2434b7 to 73140e1 Compare October 1, 2019 05:33

ashvina requested a review from goiri October 1, 2019 16:47

goiri reviewed Oct 2, 2019

View reviewed changes

...tools/hadoop-fs2img/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSTreeWalk.java Outdated Show resolved Hide resolved

goiri reviewed Oct 2, 2019

View reviewed changes

...tools/hadoop-fs2img/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSTreeWalk.java Outdated Show resolved Hide resolved

goiri approved these changes Oct 3, 2019

View reviewed changes

virajith requested changes Oct 10, 2019

View reviewed changes

...tools/hadoop-fs2img/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSTreeWalk.java Outdated Show resolved Hide resolved

...tools/hadoop-fs2img/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSTreeWalk.java Outdated Show resolved Hide resolved

goiri changed the title ~~HDFS-14856 Fetch file ACLs while mounting external store~~ HDFS-14856. Fetch file ACLs while mounting external store. Oct 10, 2019

virajith approved these changes Oct 11, 2019

View reviewed changes

ashvina added 7 commits October 11, 2019 09:31

Fetch ACL while scanning external store

ebb7286

Add unit test to verify remote acl import

7ad6623

Address review comments

04d0a8b

Make name of acl-import config consistent with the rest

85c8328

Refactor code based on review comments

84df44c

Refactor common code and exception handling

e12a30f

Add unit test to verify ugi resolution

5f1b53d

ashvina added 4 commits October 11, 2019 09:31

Fix checkstyle issues

3afebac

Terminate mounting if desired acls are unavailable

c148e4e

Fix checkstyle issues

f2f3287

Add unit test for image generation tool with acls

1bd5ee0

ashvina force-pushed the fetchRemoteAcls branch from d2c8ba4 to 1bd5ee0 Compare October 11, 2019 16:31

Fix minor checkstyle issue

0a3d262

virajith merged commit fabd41f into apache:trunk Oct 14, 2019

amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019

HDFS-14856. Fetch file ACLs while mounting external store. (apache#1478)

a8ef227

RogPodge pushed a commit to RogPodge/hadoop that referenced this pull request Mar 25, 2020

HDFS-14856. Fetch file ACLs while mounting external store. (apache#1478)

e6a5cde

HDFS-14856. Fetch file ACLs while mounting external store. #1478

HDFS-14856. Fetch file ACLs while mounting external store. #1478

Uh oh!

Conversation

ashvina commented Sep 19, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Sep 19, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Sep 21, 2019

Uh oh!

hadoop-yetus commented Sep 21, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Sep 24, 2019

Uh oh!

hadoop-yetus commented Sep 24, 2019

Uh oh!

hadoop-yetus commented Sep 24, 2019

Uh oh!

virajith left a comment

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Sep 26, 2019

Uh oh!

hadoop-yetus commented Oct 1, 2019

Uh oh!