HDDS-2187. ozone-mr test fails with No FileSystem for scheme o3fs #1537

adoroszlai · 2019-09-27T06:47:33Z

What changes were proposed in this pull request?

MR acceptance tests are failing at the Create user dir for hadoop step of Create directories required for MR test suite. This happens since HDDS-2101 removed fs.o3fs.impl definition from core-site.xml, filesystem provider is defined in META-INF of hadoop-ozone-filesystem-lib-*.jar. The problem is that the filesystem jars are not on the classpath for ozone fs commands used to create hadoop user dir in the ozone-mr env, so the explicit config is needed. This change adds back the config, but only for scm, not for Hadoop. It also applies both changes (this fix and the removal per HDDS-2101) to ozonesecure-mr env. It is based on top of #1533 (which should be merged first).

https://issues.apache.org/jira/browse/HDDS-2187

How was this patch tested?

Ran all affected acceptance tests locally.

0 datanode is up and healthy (until now)
3 datanodes are up and registered to the scm
==============================================================================
hadoop32-createmrenv :: Create directories required for MR test               
==============================================================================
Create test volume, bucket and key                                    | PASS |
------------------------------------------------------------------------------
Create user dir for hadoop                                            | PASS |
------------------------------------------------------------------------------
hadoop32-createmrenv :: Create directories required for MR test       | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Output:  /tmp/smoketest/hadoop32/result/robot-hadoop32-hadoop32-createmrenv-scm.xml
==============================================================================
hadoop32-hadoopo3fs :: Test ozone fs with hadoopfs                            
==============================================================================
Test hadoop dfs                                                       | PASS |
------------------------------------------------------------------------------
hadoop32-hadoopo3fs :: Test ozone fs with hadoopfs                    | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  /tmp/smoketest/hadoop32/result/robot-hadoop32-hadoop32-hadoopo3fs-rm.xml
==============================================================================
hadoop32-mapreduce :: Execute MR jobs                                         
==============================================================================
Execute PI calculation                                                | PASS |
------------------------------------------------------------------------------
Execute WordCount                                                     | PASS |
------------------------------------------------------------------------------
hadoop32-mapreduce :: Execute MR jobs                                 | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Output:  /tmp/smoketest/hadoop32/result/robot-hadoop32-hadoop32-mapreduce-rm.xml
Robot framework is not installed, the reports can be generated (sudo pip install robotframework).
0 datanode is up and healthy (until now)
3 datanodes are up and registered to the scm
==============================================================================
hadoop31-createmrenv :: Create directories required for MR test               
==============================================================================
Create test volume, bucket and key                                    | PASS |
------------------------------------------------------------------------------
Create user dir for hadoop                                            | PASS |
------------------------------------------------------------------------------
hadoop31-createmrenv :: Create directories required for MR test       | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Output:  /tmp/smoketest/hadoop31/result/robot-hadoop31-hadoop31-createmrenv-scm.xml
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
(1/7) Installing libbz2 (1.0.6-r7)
(2/7) Installing libffi (3.2.1-r4)
(3/7) Installing gdbm (1.13-r1)
(4/7) Installing sqlite-libs (3.25.3-r1)
(5/7) Installing python2 (2.7.15-r2)
(6/7) Installing py-setuptools (39.1.0-r0)
(7/7) Installing py2-pip (10.0.1-r0)
Executing busybox-1.28.4-r0.trigger
Executing glibc-bin-2.27-r0.trigger
OK: 93 MiB in 45 packages
Collecting robotframework
  Downloading https://files.pythonhosted.org/packages/22/0f/1b9ffa0c4e59789b50e6034866e823b7d4a5c7eaedad7bfd0bba42f2aa9d/robotframework-3.1.2-py2.py3-none-any.whl (602kB)
Installing collected packages: robotframework
Successfully installed robotframework-3.1.2
==============================================================================
hadoop31-hadoopo3fs :: Test ozone fs with hadoopfs                            
==============================================================================
Test hadoop dfs                                                       | PASS |
------------------------------------------------------------------------------
hadoop31-hadoopo3fs :: Test ozone fs with hadoopfs                    | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  /tmp/smoketest/hadoop31/result/robot-hadoop31-hadoop31-hadoopo3fs-rm.xml
==============================================================================
hadoop31-mapreduce :: Execute MR jobs                                         
==============================================================================
Execute PI calculation                                                | PASS |
------------------------------------------------------------------------------
Execute WordCount                                                     | PASS |
------------------------------------------------------------------------------
hadoop31-mapreduce :: Execute MR jobs                                 | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Output:  /tmp/smoketest/hadoop31/result/robot-hadoop31-hadoop31-mapreduce-rm.xml
Robot framework is not installed, the reports can be generated (sudo pip install robotframework).
2 datanode is up and healthy (until now)
3 datanodes are up and registered to the scm
==============================================================================
hadoop27-createmrenv :: Create directories required for MR test               
==============================================================================
Create test volume, bucket and key                                    | PASS |
------------------------------------------------------------------------------
Create user dir for hadoop                                            | PASS |
------------------------------------------------------------------------------
hadoop27-createmrenv :: Create directories required for MR test       | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Output:  /tmp/smoketest/hadoop27/result/robot-hadoop27-hadoop27-createmrenv-scm.xml
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
(1/7) Installing libbz2 (1.0.6-r7)
(2/7) Installing libffi (3.2.1-r4)
(3/7) Installing gdbm (1.13-r1)
(4/7) Installing sqlite-libs (3.25.3-r1)
(5/7) Installing python2 (2.7.15-r2)
(6/7) Installing py-setuptools (39.1.0-r0)
(7/7) Installing py2-pip (10.0.1-r0)
Executing busybox-1.28.4-r0.trigger
Executing glibc-bin-2.27-r0.trigger
OK: 93 MiB in 45 packages
Collecting robotframework
  Downloading https://files.pythonhosted.org/packages/22/0f/1b9ffa0c4e59789b50e6034866e823b7d4a5c7eaedad7bfd0bba42f2aa9d/robotframework-3.1.2-py2.py3-none-any.whl (602kB)
Installing collected packages: robotframework
Successfully installed robotframework-3.1.2
==============================================================================
hadoop27-hadoopo3fs :: Test ozone fs with hadoopfs                            
==============================================================================
Test hadoop dfs                                                       | PASS |
------------------------------------------------------------------------------
hadoop27-hadoopo3fs :: Test ozone fs with hadoopfs                    | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  /tmp/smoketest/hadoop27/result/robot-hadoop27-hadoop27-hadoopo3fs-rm.xml
==============================================================================
hadoop27-mapreduce :: Execute MR jobs                                         
==============================================================================
Execute PI calculation                                                | PASS |
------------------------------------------------------------------------------
Execute WordCount                                                     | PASS |
------------------------------------------------------------------------------
hadoop27-mapreduce :: Execute MR jobs                                 | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Output:  /tmp/smoketest/hadoop27/result/robot-hadoop27-hadoop27-mapreduce-rm.xml
Robot framework is not installed, the reports can be generated (sudo pip install robotframework).
0 datanode is up and healthy (until now)
0 datanode is up and healthy (until now)
0 datanode is up and healthy (until now)
3 datanodes are up and registered to the scm
==============================================================================
ozonesecure-mr-kinit :: Kinit test user                                       
==============================================================================
Kinit                                                                 | PASS |
------------------------------------------------------------------------------
ozonesecure-mr-kinit :: Kinit test user                               | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  /tmp/smoketest/ozonesecure-mr/result/robot-ozonesecure-mr-ozonesecure-mr-kinit-om.xml
==============================================================================
ozonesecure-mr-createmrenv :: Create directories required for MR test         
==============================================================================
Create test volume, bucket and key                                    | PASS |
------------------------------------------------------------------------------
Create user dir for hadoop                                            | PASS |
------------------------------------------------------------------------------
ozonesecure-mr-createmrenv :: Create directories required for MR test | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Output:  /tmp/smoketest/ozonesecure-mr/result/robot-ozonesecure-mr-ozonesecure-mr-createmrenv-om.xml
==============================================================================
ozonesecure-mr-kinit-hadoop :: Kinit test user                                
==============================================================================
Kinit                                                                 | PASS |
------------------------------------------------------------------------------
ozonesecure-mr-kinit-hadoop :: Kinit test user                        | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  /tmp/smoketest/ozonesecure-mr/result/robot-ozonesecure-mr-ozonesecure-mr-kinit-hadoop-rm.xml
==============================================================================
ozonesecure-mr-mapreduce :: Execute MR jobs                                   
==============================================================================
Execute PI calculation                                                | PASS |
------------------------------------------------------------------------------
Execute WordCount                                                     | PASS |
------------------------------------------------------------------------------
ozonesecure-mr-mapreduce :: Execute MR jobs                           | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================
Output:  /tmp/smoketest/ozonesecure-mr/result/robot-ozonesecure-mr-ozonesecure-mr-mapreduce-rm.xml
Robot framework is not installed, the reports can be generated (sudo pip install robotframework).

adoroszlai · 2019-09-27T06:48:23Z

/label ozone

hadoop-yetus · 2019-09-27T07:30:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	165	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
0	yamllint	0	yamllint was not available.
0	shelldocs	0	Shelldocs was not available.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
-1	mvninstall	36	hadoop-hdds in trunk failed.
-1	mvninstall	35	hadoop-ozone in trunk failed.
-1	compile	25	hadoop-hdds in trunk failed.
-1	compile	17	hadoop-ozone in trunk failed.
+1	mvnsite	0	trunk passed
+1	shadedclient	931	branch has no errors when building and testing our client artifacts.
-1	javadoc	25	hadoop-hdds in trunk failed.
-1	javadoc	21	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
-1	mvninstall	37	hadoop-hdds in the patch failed.
-1	mvninstall	30	hadoop-ozone in the patch failed.
-1	compile	28	hadoop-hdds in the patch failed.
-1	compile	24	hadoop-ozone in the patch failed.
-1	javac	28	hadoop-hdds in the patch failed.
-1	javac	24	hadoop-ozone in the patch failed.
+1	mvnsite	0	the patch passed
+1	shellcheck	1	There were no new shellcheck issues.
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	790	patch has no errors when building and testing our client artifacts.
-1	javadoc	27	hadoop-hdds in the patch failed.
-1	javadoc	23	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	32	hadoop-hdds in the patch failed.
-1	unit	28	hadoop-ozone in the patch failed.
+1	asflicense	37	The patch does not generate ASF License warnings.
		2506

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/1/artifact/out/Dockerfile
GITHUB PR	#1537
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient yamllint shellcheck shelldocs
uname	Linux e960515db5d4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `9bf7a6e`
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/1/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/1/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/1/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/1/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/1/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/1/artifact/out/branch-javadoc-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/1/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/1/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/1/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/1/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/1/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/1/artifact/out/patch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/1/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/1/artifact/out/patch-javadoc-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/1/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/1/testReport/
Max. process+thread count	411 (vs. ulimit of 5500)
modules	C: hadoop-ozone/dist U: hadoop-ozone/dist
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/1/console
versions	git=2.7.4 maven=3.3.9 shellcheck=0.4.6
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

elek · 2019-09-27T10:36:09Z

Thank you very much @adoroszlai to investigate this problem.

Fix me If I am wrong but it means that we can't use ozone fs command without adding this configuration.

Is there any option to add this file to the classpath of ozone fs (add a similar META-INF/services record to the tools project???). I think it would improve the usability.

adoroszlai · 2019-09-30T08:56:10Z

Fix me If I am wrong but it means that we can't use ozone fs command without adding this configuration.

Correct. (I thought that was known behavior.)

Is there any option to add this file to the classpath of ozone fs (add a similar META-INF/services record to the tools project???). I think it would improve the usability.

Thanks for the idea. I've updated the patch and ran some tests (Hadoop 3.1 and 2.7) locally. Let's see if CI finds any issues with it.

hadoop-yetus · 2019-09-30T09:28:57Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	42	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
0	yamllint	0	yamllint was not available.
0	shelldocs	0	Shelldocs was not available.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
0	mvndep	27	Maven dependency ordering for branch
-1	mvninstall	34	hadoop-hdds in trunk failed.
-1	mvninstall	36	hadoop-ozone in trunk failed.
-1	compile	24	hadoop-hdds in trunk failed.
-1	compile	17	hadoop-ozone in trunk failed.
+1	mvnsite	0	trunk passed
+1	shadedclient	779	branch has no errors when building and testing our client artifacts.
-1	javadoc	26	hadoop-hdds in trunk failed.
-1	javadoc	23	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
0	mvndep	22	Maven dependency ordering for patch
-1	mvninstall	40	hadoop-hdds in the patch failed.
-1	mvninstall	43	hadoop-ozone in the patch failed.
-1	compile	27	hadoop-hdds in the patch failed.
-1	compile	24	hadoop-ozone in the patch failed.
-1	javac	27	hadoop-hdds in the patch failed.
-1	javac	24	hadoop-ozone in the patch failed.
+1	mvnsite	0	the patch passed
+1	shellcheck	0	There were no new shellcheck issues.
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	732	patch has no errors when building and testing our client artifacts.
-1	javadoc	26	hadoop-hdds in the patch failed.
-1	javadoc	23	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	32	hadoop-hdds in the patch failed.
-1	unit	30	hadoop-ozone in the patch failed.
+1	asflicense	37	The patch does not generate ASF License warnings.
		2238

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/2/artifact/out/Dockerfile
GITHUB PR	#1537
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient yamllint shellcheck shelldocs
uname	Linux 26c92c2b2aab 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `760b523`
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/2/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/2/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/2/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/2/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/2/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/2/artifact/out/branch-javadoc-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/2/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/2/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/2/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/2/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/2/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/2/artifact/out/patch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/2/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/2/artifact/out/patch-javadoc-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/2/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/2/testReport/
Max. process+thread count	411 (vs. ulimit of 5500)
modules	C: hadoop-ozone/dist hadoop-ozone/tools U: hadoop-ozone
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/2/console
versions	git=2.7.4 maven=3.3.9 shellcheck=0.4.6
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-09-30T15:16:17Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	44	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
0	yamllint	0	yamllint was not available.
0	shelldocs	0	Shelldocs was not available.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
0	mvndep	16	Maven dependency ordering for branch
-1	mvninstall	37	hadoop-hdds in trunk failed.
-1	mvninstall	40	hadoop-ozone in trunk failed.
-1	compile	23	hadoop-hdds in trunk failed.
-1	compile	20	hadoop-ozone in trunk failed.
+1	mvnsite	0	trunk passed
+1	shadedclient	817	branch has no errors when building and testing our client artifacts.
-1	javadoc	25	hadoop-hdds in trunk failed.
-1	javadoc	24	hadoop-ozone in trunk failed.
		_ Patch Compile Tests _
0	mvndep	22	Maven dependency ordering for patch
-1	mvninstall	40	hadoop-hdds in the patch failed.
-1	mvninstall	41	hadoop-ozone in the patch failed.
-1	compile	30	hadoop-hdds in the patch failed.
-1	compile	24	hadoop-ozone in the patch failed.
-1	javac	30	hadoop-hdds in the patch failed.
-1	javac	24	hadoop-ozone in the patch failed.
+1	mvnsite	0	the patch passed
+1	shellcheck	0	There were no new shellcheck issues.
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	761	patch has no errors when building and testing our client artifacts.
-1	javadoc	27	hadoop-hdds in the patch failed.
-1	javadoc	23	hadoop-ozone in the patch failed.
		_ Other Tests _
-1	unit	30	hadoop-hdds in the patch failed.
-1	unit	33	hadoop-ozone in the patch failed.
+1	asflicense	39	The patch does not generate ASF License warnings.
		2291

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/Dockerfile
GITHUB PR	#1537
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient yamllint shellcheck shelldocs
uname	Linux 96c3e820363b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `b46d823`
Default Java	1.8.0_222
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/branch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/branch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/branch-javadoc-hadoop-ozone.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/patch-compile-hadoop-hdds.txt
compile	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/patch-compile-hadoop-ozone.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/patch-compile-hadoop-hdds.txt
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/patch-compile-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/patch-javadoc-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/testReport/
Max. process+thread count	413 (vs. ulimit of 5500)
modules	C: hadoop-ozone/dist hadoop-ozone/tools U: hadoop-ozone
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1537/3/console
versions	git=2.7.4 maven=3.3.9 shellcheck=0.4.6
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

elek · 2019-09-30T17:58:21Z

Thanks the update @adoroszlai Looks good. Good go have ozone fs fixed

One problem:
datanodes in ozonesecure-mr are not started AFAIK

https://github.com/elek/ozone-ci-q4/blob/master/pr/pr-hdds-2187-2nl4x/acceptance/output.log

adoroszlai · 2019-09-30T18:27:32Z

datanodes in ozonesecure-mr are not started AFAIK

https://github.com/elek/ozone-ci-q4/blob/master/pr/pr-hdds-2187-2nl4x/acceptance/output.log

I think it's ozone-recon, not ozonesecure-mr:

https://github.com/elek/ozone-ci-q4/blob/c4e58095c9584185c75d6c6af4721c33edba854a/pr/pr-hdds-2187-2nl4x/acceptance/output.log#L1453

I've seen this problem intermittently in earlier runs, eg:

https://github.com/elek/ozone-ci-q4/blob/f700ebb2254527527960593496f15008245094d6/trunk/trunk-nightly-extra-20190930-74rp4/acceptance/output.log#L3765-L3768

Working on a change to collect the docker logs.

elek · 2019-10-01T15:41:47Z

/retest

elek · 2019-10-01T15:41:59Z

If it's intermittent, let's try to retest.

adoroszlai · 2019-10-01T20:43:44Z

If it's intermittent, let's try to retest.

See HDDS-2225 for the root cause.

elek · 2019-10-02T10:51:44Z

See HDDS-2225 for the root cause.

Ok. As we understand what is this intermittent problem (and it's unrelated), let me commit this one.

adoroszlai · 2019-10-02T11:06:57Z

Thanks @elek for reviewing and committing it.

Closes apache#1537

elek added the ozone label Sep 27, 2019

adoroszlai mentioned this pull request Sep 27, 2019

HDDS-2185. createmrenv failure not reflected in acceptance test result #1533

Closed

adoroszlai force-pushed the HDDS-2187 branch from 946853e to 392de0a Compare September 30, 2019 08:50

adoroszlai added 2 commits September 30, 2019 16:36

HDDS-2187. ozone-mr test fails with No FileSystem for scheme o3fs

d99d6c8

HDDS-2187. Define FileSystem service for Ozone Tools

0aca2b2

adoroszlai force-pushed the HDDS-2187 branch from 392de0a to 0aca2b2 Compare September 30, 2019 14:37

adoroszlai mentioned this pull request Oct 2, 2019

HDDS-2225. SCM fails to start in most unsecure environments due to leftover secure config #1568

Closed

elek closed this in f1ba9bf Oct 2, 2019

adoroszlai deleted the HDDS-2187 branch October 2, 2019 11:05

amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019

HDDS-2187. ozone-mr test fails with No FileSystem for scheme "o3fs"

7690841

Closes apache#1537

RogPodge pushed a commit to RogPodge/hadoop that referenced this pull request Mar 25, 2020

HDDS-2187. ozone-mr test fails with No FileSystem for scheme "o3fs"

a25ec53

Closes apache#1537

Uh oh!

HDDS-2187. ozone-mr test fails with No FileSystem for scheme o3fs #1537

HDDS-2187. ozone-mr test fails with No FileSystem for scheme o3fs #1537

Uh oh!

Conversation

adoroszlai commented Sep 27, 2019

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

adoroszlai commented Sep 27, 2019

Uh oh!

hadoop-yetus commented Sep 27, 2019

Uh oh!

elek commented Sep 27, 2019

Uh oh!

adoroszlai commented Sep 30, 2019

Uh oh!

hadoop-yetus commented Sep 30, 2019

Uh oh!

hadoop-yetus commented Sep 30, 2019

Uh oh!

elek commented Sep 30, 2019

Uh oh!

adoroszlai commented Sep 30, 2019

Uh oh!

elek commented Oct 1, 2019

Uh oh!

elek commented Oct 1, 2019

Uh oh!

adoroszlai commented Oct 1, 2019

Uh oh!

elek commented Oct 2, 2019

Uh oh!

adoroszlai commented Oct 2, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants