MAPREDUCE-7428. Fix failing MapReduce tests due to the JUnit upgrades in WebServicesTestUtils #5243

aajisaka · 2022-12-19T16:43:24Z

Description of PR

Remove direct JUnit API usages from WebServiceTestUtils to fix MapReduce test failures caused by YARN-6946. In addition, it reverts #5209 to run JUnit 4 based tests in hadoop-mapreduce-client-app module.

How was this patch tested?

Ran TestAMWebServices and TestHsWebServices successfully on my local.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
n/a Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
n/a If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
n/a If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

…preduce-client-hs

…adoop-mapreduce-client-hs" This reverts commit f7e550d.

…grade in org.apache.hadoop.mapreduce.v2.app.webapp (apache#5209)" This reverts commit 85ec796.

slfan1989 · 2022-12-20T05:04:00Z

@steveloughran @aajisaka @ashutoshcipher I read this part of the code, I try to submit a pr(#5245), I think if we roll back yarn's junit upgrade, some pity. I will add a WebServicesTestUtils.java separately in the hadoop-mapreduce-client-app module, and modify the reference, which seems better. We wait for the result of yetus.

I hope this pr(#5245) to solve the mapreduce junit-test problem.

slfan1989 · 2022-12-20T09:47:50Z

@aajisaka I tried modifying the code and I found yours to be the best, LGTM.
@steveloughran I think this part of the code can be merged to the trunk branch, This can solve mr's junit test error problem.

The reason for the problem is that the hadoop-mapreduce-client-hs and hadoop-mapreduce-client-app modules referenced the hadoop-yarn-common test method, and the hadoop-yarn-common module upgraded the junit-test package, So there is an error of class not found.

hotcodemacha · 2022-12-20T09:50:01Z

Thanks @aajisaka for taking of this. Sorry I was not around for last few days.

hotcodemacha

LGTM

steveloughran · 2022-12-20T11:49:55Z

...-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestContainerLogsUtils.java

    }
-    assertTrue(fs.mkdirs(appLogsDir));
-
+    fs.mkdirs(appLogsDir);


this does cut the check of the outcome...are we all happy with this?

I thought we are happy because the directory has been removed in the previous line, but rethinking this, it's possible to fail removing the directory and then mkdir fails. Adding back the check in the next commit.

Thanks a lot for your question, I think we should keep this assertTrue.

We are currently experiencing some difficulties.

TestContainerLogsUtils.java uses assertTrue of org.junit.jupiter, hadoop-mapreduce-client-hs and hadoop-mapreduce-client-app use TestContainerLogsUtils.java, hadoop-mapreduce-client-hs and hadoop-mapreduce-client-app These two modules have not upgraded the junit test, so when running the test, the classes cannot be found.

hadoop-mapreduce-client-hs and hadoop-mapreduce-client-app may take some time to upgrade junit test.

aajisaka's idea can quickly fix this problem without affecting the compilation of other modules.

I will continue to follow up the mapreduce junit upgrade. If the mapreduce module junit upgrade is completed, I will change this back.

@aajisaka I see your latest changes, your changes are reasonable. LGTM.

I thought we are happy because the directory has been removed in the previous line, but rethinking this, it's possible to fail removing the directory and then mkdir fails. Adding back the check in the next commit.

Thanks for adding it back. It makes sense. LGTM

Thanks @slfan1989

hadoop-mapreduce-client-hs and hadoop-mapreduce-client-app may take some time to upgrade junit test.

I have already raised PR for both the modules. Can see some failures in hadoop-mapreduce-client-app - will fix it as well in that PR.

how about removing the asserts and just calling getFileStatus(dir) after.

remember also that delete() returning false means "dir wasn't there"...not a reason to fail

Added getFileStatus(dir) to check the dir exists in the latest commit. Thank you @steveloughran

hadoop-yetus · 2022-12-20T16:08:48Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 55s		Maven dependency ordering for branch
-1 ❌	mvninstall	17m 27s	/branch-mvninstall-root.txt	root in trunk failed.
+1 💚	compile	24m 42s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	21m 50s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	checkstyle	4m 0s		trunk passed
+1 💚	mvnsite	2m 26s		trunk passed
-1 ❌	javadoc	0m 43s	/branch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-mapreduce-client-app in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
-1 ❌	javadoc	0m 40s	/branch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-mapreduce-client-hs in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	2m 21s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	4m 17s		trunk passed
+1 💚	shadedclient	20m 51s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 32s		the patch passed
+1 💚	compile	22m 19s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	22m 19s		the patch passed
+1 💚	compile	20m 28s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	20m 28s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	3m 35s		the patch passed
+1 💚	mvnsite	2m 45s		the patch passed
-1 ❌	javadoc	0m 48s	/patch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-mapreduce-client-app in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
-1 ❌	javadoc	0m 45s	/patch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-mapreduce-client-hs in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	2m 25s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	4m 46s		the patch passed
+1 💚	shadedclient	20m 54s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	5m 47s		hadoop-yarn-common in the patch passed.
+1 💚	unit	8m 47s		hadoop-mapreduce-client-app in the patch passed.
+1 💚	unit	5m 37s		hadoop-mapreduce-client-hs in the patch passed.
+1 💚	asflicense	1m 2s		The patch does not generate ASF License warnings.
		223m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5243/4/artifact/out/Dockerfile
GITHUB PR	#5243
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname	Linux d5fffa3260f6 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e1269ae`
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5243/4/testReport/
Max. process+thread count	722 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5243/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-12-23T13:26:59Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 4s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 16s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 37s		trunk passed
+1 💚	compile	23m 25s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	20m 38s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	checkstyle	3m 46s		trunk passed
+1 💚	mvnsite	2m 50s		trunk passed
-1 ❌	javadoc	0m 49s	/branch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-mapreduce-client-app in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
-1 ❌	javadoc	0m 44s	/branch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-mapreduce-client-hs in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	2m 28s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	4m 24s		trunk passed
+1 💚	shadedclient	20m 58s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 32s		the patch passed
+1 💚	compile	22m 24s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	22m 24s		the patch passed
+1 💚	compile	20m 35s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	20m 35s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	3m 39s		the patch passed
+1 💚	mvnsite	2m 45s		the patch passed
-1 ❌	javadoc	0m 47s	/patch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-mapreduce-client-app in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
-1 ❌	javadoc	0m 45s	/patch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-mapreduce-client-hs in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	2m 31s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	4m 47s		the patch passed
+1 💚	shadedclient	20m 57s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	5m 48s		hadoop-yarn-common in the patch passed.
+1 💚	unit	8m 53s		hadoop-mapreduce-client-app in the patch passed.
+1 💚	unit	5m 31s		hadoop-mapreduce-client-hs in the patch passed.
+1 💚	asflicense	1m 3s		The patch does not generate ASF License warnings.
		231m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5243/5/artifact/out/Dockerfile
GITHUB PR	#5243
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname	Linux cc263e525e93 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ff6d5fa`
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5243/5/testReport/
Max. process+thread count	716 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5243/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran

LGTM
+1

ayushtkn

LGTM
Thanx @aajisaka for helping chasing and fixing the tests :-)

aajisaka · 2022-12-23T19:34:10Z

Thanks! Merged into trunk.

aajisaka added 4 commits December 20, 2022 01:37

Test: Remove JUnit API from WebServicesTestUtils to fix MAPREDUCE-7428

baffc92

Add changes to run tests in hadoop-mapreduce-client-app and hadoop-ma…

f7e550d

…preduce-client-hs

Revert "Add changes to run tests in hadoop-mapreduce-client-app and h…

31773be

…adoop-mapreduce-client-hs" This reverts commit f7e550d.

Revert "MAPREDUCE-7428. Fix failures related to Junit 4 to Junit 5 up…

2889872

…grade in org.apache.hadoop.mapreduce.v2.app.webapp (apache#5209)" This reverts commit 85ec796.

aajisaka changed the title ~~Draft: Fix failing MapReduce tests due to the JUnit upgrades in WebServicesTestUtils~~ MAPREDUCE-7428. Fix failing MapReduce tests due to the JUnit upgrades in WebServicesTestUtils Dec 19, 2022

aajisaka marked this pull request as ready for review December 19, 2022 16:55

This comment was marked as outdated.

Sign in to view

Remove JUnit API from TestContainerLogsUtils

75f9604

This comment was marked as outdated.

Sign in to view

slfan1989 approved these changes Dec 20, 2022

View reviewed changes

hotcodemacha approved these changes Dec 20, 2022

View reviewed changes

steveloughran reviewed Dec 20, 2022

View reviewed changes

steveloughran mentioned this pull request Dec 20, 2022

MAPREDUCE-7422/MAPREDUCE-7428 #5239

Closed

Add check

e1269ae

slfan1989 mentioned this pull request Dec 22, 2022

MAPREDUCE-7429 Application log link is not accessible via TimelineServer WebUI in IPV6 scenario #5236

Open

Remove AssertJ libraries

ff6d5fa

steveloughran approved these changes Dec 23, 2022

View reviewed changes

ayushtkn approved these changes Dec 23, 2022

View reviewed changes

aajisaka merged commit 049d176 into apache:trunk Dec 23, 2022

aajisaka deleted the fix-junit-errors branch December 23, 2022 19:33

MAPREDUCE-7428. Fix failing MapReduce tests due to the JUnit upgrades in WebServicesTestUtils #5243

MAPREDUCE-7428. Fix failing MapReduce tests due to the JUnit upgrades in WebServicesTestUtils #5243

Uh oh!

Conversation

aajisaka commented Dec 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

slfan1989 commented Dec 20, 2022

Uh oh!

slfan1989 commented Dec 20, 2022

Uh oh!

hotcodemacha commented Dec 20, 2022

Uh oh!

hotcodemacha left a comment

Choose a reason for hiding this comment

Uh oh!

steveloughran Dec 20, 2022

Choose a reason for hiding this comment

Uh oh!

aajisaka Dec 20, 2022

Choose a reason for hiding this comment

Uh oh!

slfan1989 Dec 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

slfan1989 Dec 20, 2022

Choose a reason for hiding this comment

Uh oh!

hotcodemacha Dec 20, 2022

Choose a reason for hiding this comment

Uh oh!

hotcodemacha Dec 20, 2022

Choose a reason for hiding this comment

Uh oh!

steveloughran Dec 22, 2022

Choose a reason for hiding this comment

Uh oh!

aajisaka Dec 23, 2022

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Dec 20, 2022

Uh oh!

hadoop-yetus commented Dec 23, 2022

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

aajisaka commented Dec 23, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

aajisaka commented Dec 19, 2022 •

edited

Loading

slfan1989 Dec 20, 2022 •

edited

Loading