HADOOP-16560. [YARN] use protobuf-maven-plugin to generat… #1496

Apache9 · 2019-09-22T07:11:46Z

…e protobuf classes

Apache9 · 2019-09-22T07:12:53Z

The patch here shares some contents with #1494 , will do a rebase after #1494 is merged.

vinayakumarb

Looks like changes of YARN and MAPREDUCE together is not able to meet precommit jenkins timeout deadline of 5 hrs.
May need to separate?

hadoop-project/pom.xml

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/pom.xml

hadoop-yetus · 2019-09-23T14:03:13Z

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml

whitespace:tabs in line

hadoop-yetus · 2019-09-23T14:03:15Z

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml

whitespace:tabs in line

hadoop-yetus · 2019-09-23T14:03:16Z

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml

whitespace:tabs in line

hadoop-yetus · 2019-09-23T14:03:18Z

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml

whitespace:tabs in line

hadoop-yetus · 2019-09-23T14:03:20Z

...-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml

whitespace:tabs in line

hadoop-yetus · 2019-09-23T14:03:23Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	1376	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 3 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	41	Maven dependency ordering for branch
+1	mvninstall	1076	trunk passed
+1	compile	496	trunk passed
+1	mvnsite	441	trunk passed
+1	shadedclient	2778	branch has no errors when building and testing our client artifacts.
+1	javadoc	371	trunk passed
		_ Patch Compile Tests _
0	mvndep	19	Maven dependency ordering for patch
+1	mvninstall	306	the patch passed
+1	compile	455	the patch passed
+1	cc	455	the patch passed
-1	javac	455	hadoop-yarn-project_hadoop-yarn generated 1 new + 332 unchanged - 1 fixed = 333 total (was 333)
+1	mvnsite	402	the patch passed
-1	whitespace	0	The patch 5 line(s) with tabs.
+1	xml	12	The patch has no ill-formed XML file.
+1	shadedclient	778	patch has no errors when building and testing our client artifacts.
+1	javadoc	354	the patch passed
		_ Other Tests _
+1	unit	61	hadoop-yarn-api in the patch passed.
+1	unit	248	hadoop-yarn-common in the patch passed.
+1	unit	184	hadoop-yarn-server-common in the patch passed.
+1	unit	1322	hadoop-yarn-server-nodemanager in the patch passed.
+1	unit	225	hadoop-yarn-server-applicationhistoryservice in the patch passed.
-1	unit	4992	hadoop-yarn-server-resourcemanager in the patch failed.
+1	unit	198	hadoop-yarn-server-tests in the patch passed.
+1	unit	1608	hadoop-yarn-client in the patch passed.
+1	unit	1157	hadoop-yarn-services-core in the patch passed.
+1	asflicense	51	The patch does not generate ASF License warnings.
		17098

Reason	Tests
Failed junit tests	hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1496/2/artifact/out/Dockerfile
GITHUB PR	#1496
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml cc
uname	Linux c6cc5b62f2fd 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 07c81e9
Default Java	1.8.0_222
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1496/2/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt
whitespace	https://builds.apache.org/job/hadoop-multibranch/job/PR-1496/2/artifact/out/whitespace-tabs.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1496/2/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1496/2/testReport/
Max. process+thread count	813 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core U: hadoop-yarn-project/hadoop-yarn
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1496/2/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

…lasses

vinayakumarb · 2019-09-23T17:18:51Z

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml

-            <goals>
-              <goal>protoc</goal>
-            </goals>
+            <id>src-test-compile-protoc</id>


Before this change, YarnSecurityTestAMRMTokenProtos was getting generated in "target/generated-sources" directory and has been treated as main src. But ideally it should be part of test only.
This should not be a problem I hope.

vinayakumarb · 2019-09-23T17:20:35Z

Test failures seems to be unrelated. Please re-check.

javac error is mostly due to change in the location of the test proto file as mentioned above.

+1
pending whitespace corrections.

hadoop-yetus · 2019-09-23T18:54:59Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	41	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 3 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	18	Maven dependency ordering for branch
+1	mvninstall	1138	trunk passed
+1	compile	521	trunk passed
+1	mvnsite	423	trunk passed
+1	shadedclient	2912	branch has no errors when building and testing our client artifacts.
+1	javadoc	365	trunk passed
		_ Patch Compile Tests _
0	mvndep	21	Maven dependency ordering for patch
+1	mvninstall	350	the patch passed
+1	compile	596	the patch passed
+1	cc	596	the patch passed
-1	javac	596	hadoop-yarn-project_hadoop-yarn generated 1 new + 332 unchanged - 1 fixed = 333 total (was 333)
+1	mvnsite	409	the patch passed
+1	whitespace	1	The patch has no whitespace issues.
+1	xml	13	The patch has no ill-formed XML file.
+1	shadedclient	973	patch has no errors when building and testing our client artifacts.
+1	javadoc	341	the patch passed
		_ Other Tests _
+1	unit	63	hadoop-yarn-api in the patch passed.
+1	unit	328	hadoop-yarn-common in the patch passed.
+1	unit	211	hadoop-yarn-server-common in the patch passed.
+1	unit	1373	hadoop-yarn-server-nodemanager in the patch passed.
+1	unit	236	hadoop-yarn-server-applicationhistoryservice in the patch passed.
+1	unit	5057	hadoop-yarn-server-resourcemanager in the patch passed.
+1	unit	200	hadoop-yarn-server-tests in the patch passed.
+1	unit	1599	hadoop-yarn-client in the patch passed.
+1	unit	1145	hadoop-yarn-services-core in the patch passed.
+1	asflicense	51	The patch does not generate ASF License warnings.
		16485

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1496/3/artifact/out/Dockerfile
GITHUB PR	#1496
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml cc
uname	Linux baeffaa05d44 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `2b5fc95`
Default Java	1.8.0_222
javac	https://builds.apache.org/job/hadoop-multibranch/job/PR-1496/3/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1496/3/testReport/
Max. process+thread count	821 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core U: hadoop-yarn-project/hadoop-yarn
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1496/3/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Apache9 · 2019-09-24T00:48:46Z

The javadoc warning is from generated source? I believe we should skip the checks on the generated files?

vinayakumarb · 2019-09-24T06:14:46Z

The javadoc warning is from generated source? I believe we should skip the checks on the generated files?

Yes, there are many javac warnings are coming from generate source. Guess some API is deprecated, but generator still uses them. (may be because of syntax=proto2).
I guess that's fine. Just need to make sure, we dont use any of these deprecated code in other sources.

…lasses (apache#1496) Signed-off-by: Vinayakumar B <[email protected]>

Apache9 self-assigned this Sep 22, 2019

vinayakumarb reviewed Sep 22, 2019

View reviewed changes

Apache9 mentioned this pull request Sep 23, 2019

HADOOP-16558. [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes #1494

Merged

Apache9 force-pushed the HADOOP-16560 branch from c34dd2e to 8d1a87e Compare September 23, 2019 08:55

Apache9 changed the title ~~HADOOP-16560. [YARN + MAPREDUCE] use protobuf-maven-plugin to generat…~~ HADOOP-16560. [YARN] use protobuf-maven-plugin to generat… Sep 23, 2019