HADOOP-17472. Upgrade hadoop default version in hadoop.tools.dynamometer.TestDynamometerInfra #2622

viirya · 2021-01-15T04:57:29Z

See https://issues.apache.org/jira/browse/HADOOP-17472 for details.

The CI found a failed test hadoop.tools.dynamometer.TestDynamometerInfra in #2611.

Seems currently hadoop.tools.dynamometer.TestDynamometerInfra in the trunk is failed.

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.tools.dynamometer.TestDynamometerInfra
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.531 s <<< FAILURE! - in org.apache.hadoop.tools.dynamometer.TestDynamometerInfra
[ERROR] org.apache.hadoop.tools.dynamometer.TestDynamometerInfra  Time elapsed: 0.53 s  <<< ERROR!
java.io.FileNotFoundException: http://mirrors.ocf.berkeley.edu/apache/hadoop/common/hadoop-3.1.3/hadoop-3.1.3.tar.gz
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1923)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1523)
        at org.apache.commons.io.FileUtils.copyURLToFile(FileUtils.java:1506)
        at org.apache.hadoop.tools.dynamometer.DynoInfraUtils.fetchHadoopTarball(DynoInfraUtils.java:151)
        at org.apache.hadoop.tools.dynamometer.TestDynamometerInfra.setupClass(TestDynamometerInfra.java:176)

There is no more 3.1.3 in http://mirrors.ocf.berkeley.edu/apache/hadoop/common/. So I guess we need to upgrade it to 3.1.4.

viirya · 2021-01-15T04:58:33Z

cc @sunchao

hadoop-yetus · 2021-01-15T06:59:33Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	32m 43s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 56s		trunk passed
+1 💚	compile	0m 30s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	0m 22s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	0m 20s		trunk passed
+1 💚	mvnsite	0m 26s		trunk passed
+1 💚	shadedclient	18m 4s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 23s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 22s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	0m 41s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 39s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 24s		the patch passed
+1 💚	compile	0m 19s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	0m 19s		the patch passed
+1 💚	compile	0m 17s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	0m 17s		the patch passed
+1 💚	checkstyle	0m 12s		the patch passed
+1 💚	mvnsite	0m 18s		the patch passed
+1 💚	whitespace	0m 1s		The patch has no whitespace issues.
+1 💚	shadedclient	17m 4s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 19s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 18s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	0m 42s		the patch passed
			_ Other Tests _
-1 ❌	unit	8m 12s	/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-infra.txt	hadoop-dynamometer-infra in the patch passed.
+1 💚	asflicense	0m 31s		The patch does not generate ASF License warnings.
		120m 56s

Reason	Tests
Failed junit tests	hadoop.tools.dynamometer.TestDynamometerInfra

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2622/1/artifact/out/Dockerfile
GITHUB PR	#2622
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 8015fabbb82b 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `630f8dd`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2622/1/testReport/
Max. process+thread count	932 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra U: hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2622/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

viirya · 2021-01-15T07:08:16Z

Got another error now:

[INFO] Running org.apache.hadoop.tools.dynamometer.TestDynamometerInfra
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 469.118 s <<< FAILURE! - in org.apache.hadoop.tools.dynamometer.TestDynamometerInfra
[ERROR] testNameNodeInYARN(org.apache.hadoop.tools.dynamometer.TestDynamometerInfra)  Time elapsed: 298.662 s  <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<6>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:645)
	at org.junit.Assert.assertEquals(Assert.java:631)
	at org.apache.hadoop.tools.dynamometer.TestDynamometerInfra.testNameNodeInYARN(TestDynamometerInfra.java:379)

sunchao · 2021-01-15T18:02:32Z

There are more test failures even after this. It seems this test has been broken for some time. cc @xkrogen .

viirya · 2021-01-19T21:06:18Z

Hmm, I manually commented the line comparing TOTALINVALIDCOMMANDS, and got java.util.concurrent.TimeoutException locally.

Is this test still valid to have?

tasanuma · 2021-01-20T12:04:48Z

This is a duplicate of #2471, but you may proceed. I also encountered the same error, but I still haven't been able to fix it.

I would recommend using archive.apache.org as the repository as in #2471.

xkrogen · 2021-01-20T16:55:15Z

Thanks for the ping @sunchao . I'll try to take a look and see if I can figure out what's wrong.

xkrogen · 2021-01-20T23:53:09Z

I haven't been able to make it this test run locally, I am getting errors like:

[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 15.284 s <<< FAILURE! - in org.apache.hadoop.tools.dynamometer.TestDynamometerInfra
[ERROR] org.apache.hadoop.tools.dynamometer.TestDynamometerInfra  Time elapsed: 15.283 s  <<< ERROR!
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.ExceptionInInitializerError
        at org.apache.hadoop.yarn.server.MiniYARNCluster.startResourceManager(MiniYARNCluster.java:378)
        at org.apache.hadoop.yarn.server.MiniYARNCluster.access$300(MiniYARNCluster.java:127)
        at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:494)
...
Caused by: java.lang.IllegalArgumentException: ReRegistration of rpcKind: RPC_PROTOCOL_BUFFER
        at org.apache.hadoop.ipc.Server.registerProtocolEngine(Server.java:290)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2.<clinit>(ProtobufRpcEngine2.java:64)

AFAICT it looks like both ProtobufRpcEngine and ProtobufRpcEngine2 are being loaded and their registrations are conflicting, but I don't know why. Looks like others haven't been hitting this issue so I'm not sure what's wrong.

Regarding TOTALINVALIDCOMMANDS, 6 is all of them, so it looks like the NameNode is starting up but something is wrong with it because all of the FS calls to it are resulting in exceptions. This is probably related to why the workload job isn't completing and so you're seeing a timeout. You should be able to check the log files for the application, which end up within the NodeManager log directories of the MiniYARNCluster -- the path is like:

./hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/target/org.apache.hadoop.tools.dynamometer.TestDynamometerInfra/org.apache.hadoop.tools.dynamometer.TestDynamometerInfra-logDir-nm*/*

This dir gets cleaned up by default when the test exits, so you may want to comment out line 296 to avoid deleting it and losing the output.

Also re: archive.apache.org, I avoided using it initially because it's not intended for much bandwidth. They have this message up on the home page, and if I remember correctly it was previously more strongly worded about always using a mirror instead:

If you are looking for current software releases, please visit one of our numerous mirrors. Do note that a daily limit of 5GB per IP is being enforced on archive.apache.org, to prevent abuse.

The right long term fix should be HDFS-14412, which will use the local Hadoop build instead of localizing it from the web... But not sure what to do in the short term.

viirya · 2021-01-23T07:56:17Z

Thanks @xkrogen. Let me check the log files to see if there is any clue.

steveloughran · 2021-03-09T19:42:27Z

With #2537 in, what is the status of this?

viirya · 2021-03-09T20:01:26Z

Does #2537 help this issue? Let me sync up with trunk.

tasanuma · 2021-04-30T16:06:11Z

Hi @viirya, are you still working on this?

tasanuma · 2021-05-04T11:46:13Z

As we will release 3.3.1 soon, I'd like to rework this issue in #2471.

tasanuma · 2021-05-07T08:16:49Z

As #2471 merged, I'm closing this PR. Thanks for your early work, @viirya and @xkrogen.

Fix version.

2311634

tasanuma mentioned this pull request Jan 21, 2021

HADOOP-17224. Install Intel ISA-L library in Dockerfile. #2537

Merged

Merge remote-tracking branch 'upstream/trunk' into HADOOP-17472

f77f8f9

tasanuma mentioned this pull request May 6, 2021

HADOOP-17375. Fix the error of TestDynamometerInfra. #2471

Merged

tasanuma closed this May 7, 2021

HADOOP-17472. Upgrade hadoop default version in hadoop.tools.dynamometer.TestDynamometerInfra #2622

HADOOP-17472. Upgrade hadoop default version in hadoop.tools.dynamometer.TestDynamometerInfra #2622

Uh oh!

Conversation

viirya commented Jan 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

viirya commented Jan 15, 2021

Uh oh!

hadoop-yetus commented Jan 15, 2021

Uh oh!

viirya commented Jan 15, 2021

Uh oh!

sunchao commented Jan 15, 2021

Uh oh!

viirya commented Jan 19, 2021

Uh oh!

tasanuma commented Jan 20, 2021

Uh oh!

xkrogen commented Jan 20, 2021

Uh oh!

xkrogen commented Jan 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

viirya commented Jan 23, 2021

Uh oh!

steveloughran commented Mar 9, 2021

Uh oh!

viirya commented Mar 9, 2021

Uh oh!

tasanuma commented Apr 30, 2021

Uh oh!

tasanuma commented May 4, 2021

Uh oh!

tasanuma commented May 7, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

viirya commented Jan 15, 2021 •

edited

Loading

xkrogen commented Jan 20, 2021 •

edited

Loading