MAPREDUCE-7294. Only application master should upload resource to Yarn Shared Cache #2223

JohnZZGithub · 2020-08-13T04:33:45Z

…esource to yarn shared cache manager.

NOTICE

Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

JohnZZGithub · 2020-08-13T04:34:10Z

https://issues.apache.org/jira/browse/YARN-10398

hadoop-yetus · 2020-08-13T08:26:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 52s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	3m 12s	Maven dependency ordering for branch
+1 💚	mvninstall	33m 43s	trunk passed
+1 💚	compile	3m 0s	trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	compile	2m 31s	trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	checkstyle	0m 57s	trunk passed
+1 💚	mvnsite	1m 30s	trunk passed
+1 💚	shadedclient	20m 39s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 57s	trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javadoc	0m 52s	trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+0 🆗	spotbugs	1m 38s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 51s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 32s	Maven dependency ordering for patch
+1 💚	mvninstall	1m 17s	the patch passed
+1 💚	compile	2m 51s	the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javac	2m 51s	the patch passed
+1 💚	compile	2m 15s	the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	javac	2m 15s	the patch passed
+1 💚	checkstyle	0m 44s	the patch passed
+1 💚	mvnsite	1m 11s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	17m 44s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 47s	the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javadoc	0m 43s	the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
+1 💚	findbugs	2m 48s	the patch passed
		_ Other Tests _
+1 💚	unit	7m 46s	hadoop-mapreduce-client-core in the patch passed.
-1 ❌	unit	23m 7s	hadoop-mapreduce-client-app in the patch passed.
-1 ❌	asflicense	0m 50s	The patch generated 2 ASF License warnings.
		135m 56s

Reason	Tests
Failed junit tests	hadoop.mapreduce.v2.app.webapp.TestAMWebServices
	hadoop.mapreduce.v2.app.TestRuntimeEstimators
	hadoop.mapreduce.v2.app.webapp.TestAMWebServicesAttempts

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2223/1/artifact/out/Dockerfile
GITHUB PR	#2223
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 7c9ebc773364 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `e592ec5`
Default Java	Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
unit	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2223/1/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2223/1/testReport/
asflicense	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2223/1/artifact/out/patch-asflicense-problems.txt
Max. process+thread count	1134 (vs. ulimit of 5500)
modules	C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2223/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

JohnZZGithub · 2020-08-18T20:13:37Z

@steveloughran Could you please help review the patch? Thanks

JohnZZGithub · 2020-08-18T20:14:19Z

https://issues.apache.org/jira/browse/YARN-10398

jiwq

Thanks @JohnZZGithub for the contribution. I found this PR's topic and commit message is not related. So I think we should create a new ticket under MAPREDUCE project and amend this commit.

steveloughran · 2020-08-22T17:33:36Z

@steveloughran Could you please help review the patch? Thanks

I don't do yarn PRs

JohnZZGithub · 2020-08-24T04:53:04Z

@jiwq I double checked and confirmed the PR is the fix for the problem. The reason why the non-application master tries to upload is that the clear cache code didn't work. The code and bug are in YARN. MR uses yarn shared cache. I'm not sure we should move it MR project. Thanks.

JohnZZGithub · 2020-08-24T04:53:22Z

@steveloughran gotcha, thanks anyway.

jiwq · 2020-08-24T15:45:56Z

@jiwq I double checked and confirmed the PR is the fix for the problem. The reason why the non-application master tries to upload is that the clear cache code didn't work. The code and bug are in YARN. MR uses yarn shared cache. I'm not sure we should move it MR project. Thanks.

@JohnZZGithub The YARN Shared Cache is used to all YARN applications, but this PR is related to the MapReduce, so I think we should move it to MAPREDUCE project.

JohnZZGithub · 2020-09-12T00:08:21Z

@liuml07 Could you please review it when you get a chance?

JohnZZGithub · 2020-09-12T00:20:56Z

...educe-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java

The bug is when the policies are empty it won't clean up the existing policies.

liuml07

+1 after comments are addressed.

A second opinion is appreciated before I commit it. CC: @jiwq @aajisaka

...educe-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java

…esource to yarn shared cache manager.

JohnZZGithub · 2020-09-15T06:54:51Z

@liuml07 Thanks a lot for the detailed review, updated the PR.

liuml07

+1 on minor change.

Failing tests were not related and can not be reproduced locally.

I'll commit this in 3 business days in case there is a second review (welcome!)

JohnZZGithub · 2020-09-16T04:28:04Z

@liuml07 Thanks a ton!

…n Shared Cache (#2223) Contributed by Zhenzhao Wang <[email protected]> Signed-off-by: Mingliang Liu <[email protected]>

…n Shared Cache (apache#2223) Contributed by Zhenzhao Wang <[email protected]> Signed-off-by: Mingliang Liu <[email protected]>

jiwq reviewed Aug 22, 2020

View reviewed changes

JohnZZGithub changed the title ~~YARN-10398. Fix the bug to make sure only application master upload resource to Yarn Shared Cache~~ MAPREDUCE-7294. Fix the bug to make sure only application master upload resource to Yarn Shared Cache Aug 24, 2020

JohnZZGithub commented Sep 12, 2020

View reviewed changes

liuml07 approved these changes Sep 12, 2020

View reviewed changes

Zhenzhaow added 2 commits September 14, 2020 23:34

YARN-10398. Fix the bug to make sure only application master upload r…

9428c1e

…esource to yarn shared cache manager.

Adress comments and rebase code

b356f23

JohnZZGithub force-pushed the YARN-10398 branch from a15e59c to b356f23 Compare September 15, 2020 06:53

JohnZZGithub requested a review from liuml07 September 15, 2020 06:54

liuml07 approved these changes Sep 15, 2020

View reviewed changes

liuml07 merged commit 95dfc87 into apache:trunk Sep 20, 2020

asfgit pushed a commit that referenced this pull request Sep 20, 2020

MAPREDUCE-7294. Only application master should upload resource to Yar…

e5e9139

…n Shared Cache (#2223) Contributed by Zhenzhao Wang <[email protected]> Signed-off-by: Mingliang Liu <[email protected]>

asfgit pushed a commit that referenced this pull request Sep 20, 2020

MAPREDUCE-7294. Only application master should upload resource to Yar…

0873555

…n Shared Cache (#2223) Contributed by Zhenzhao Wang <[email protected]> Signed-off-by: Mingliang Liu <[email protected]>

asfgit pushed a commit that referenced this pull request Sep 20, 2020

MAPREDUCE-7294. Only application master should upload resource to Yar…

06ff4d1

…n Shared Cache (#2223) Contributed by Zhenzhao Wang <[email protected]> Signed-off-by: Mingliang Liu <[email protected]>

liuml07 changed the title ~~MAPREDUCE-7294. Fix the bug to make sure only application master upload resource to Yarn Shared Cache~~ MAPREDUCE-7294. Only application master should upload resource to Yarn Shared Cache Sep 20, 2020

MAPREDUCE-7294. Only application master should upload resource to Yarn Shared Cache #2223

MAPREDUCE-7294. Only application master should upload resource to Yarn Shared Cache #2223

Uh oh!

Conversation

JohnZZGithub commented Aug 13, 2020

NOTICE

Uh oh!

JohnZZGithub commented Aug 13, 2020

Uh oh!

hadoop-yetus commented Aug 13, 2020

Uh oh!

JohnZZGithub commented Aug 18, 2020

Uh oh!

JohnZZGithub commented Aug 18, 2020

Uh oh!

jiwq left a comment

Choose a reason for hiding this comment

Uh oh!

steveloughran commented Aug 22, 2020

Uh oh!

JohnZZGithub commented Aug 24, 2020

Uh oh!

JohnZZGithub commented Aug 24, 2020

Uh oh!

jiwq commented Aug 24, 2020

Uh oh!

JohnZZGithub commented Sep 12, 2020

Uh oh!

JohnZZGithub Sep 12, 2020

Choose a reason for hiding this comment

Uh oh!

liuml07 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JohnZZGithub commented Sep 15, 2020

Uh oh!

liuml07 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JohnZZGithub commented Sep 16, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

liuml07 left a comment •

edited

Loading