HBASE-28368 Backport "HBASE-27693 Support for Hadoop's LDAP Authentic… #5680

NihalJain · 2024-02-14T12:54:23Z

…ation mechanism (Web UI only)" to branch-2
Co-authored-by: Yash Dodeja [email protected]

NihalJain · 2024-02-14T12:54:38Z

Backports #5144 / #5213

NihalJain · 2024-02-14T15:38:48Z

The test TestLogLevel fails for hadoop-2, but passes for hadoop-3. Will analyse and fix with next commit.

Apache9 · 2024-02-18T10:33:18Z

The failure for jdk8 hadoop2 is related?

NihalJain · 2024-02-18T11:03:52Z

Yes I have checked the test failures are related. There's more changes needed for hadoop2. In hadoop 2 the version of directory services used is different from what we are importing for hadoop 3 and both versions have somewhat incompatible changes. I will submit a fix to make test run for both.

NihalJain · 2025-05-21T05:58:43Z

The failure precommit checks / yetus jdk8 Hadoop2 checks / org.apache.hadoop.hbase.http.log.TestLogLevel.(?) is related:

java.lang.NoClassDefFoundError: org/apache/directory/api/ldap/schemaextractor/SchemaLdifExtractor
	at org.apache.hadoop.hbase.http.log.TestLogLevel.setupMiniKdc(TestLogLevel.java:116)
	at org.apache.hadoop.hbase.http.log.TestLogLevel.setUp(TestLogLevel.java:107)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassNotFoundException: org.apache.directory.api.ldap.schemaextractor.SchemaLdifExtractor
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 16 more

We see this as we have overridden the apacheds version in pom for all hadoop versions but for hadoop 2 it fails to start KDC as the above package does not even exist in apacheds 2.0.0.AM26 while hadoop 2 requires 2.0.0-M15.
Moving to 2.0.0-M15 causes test class to fail as annotations like CreateLdapServer etc are not supported with that version.

So as a fix I propose to skip running LDAP tests fro hadoop2 and also moving all dependency changes to be only for hadoop3.

NihalJain · 2025-05-21T11:34:50Z

Failures are not related, previous build test failure issue is resolved

ndimiduk · 2025-05-21T12:03:34Z

hbase-http/pom.xml

+            <artifactId>maven-compiler-plugin</artifactId>
+            <configuration>
+              <!--
+                Below tests fail with hadoop 2 due to test dependency versioning issues.  Hence, skip LDAP tests for Hadoop 2,


This approach is clever but quite unexpected for the average dev and thus a bit scary to me.

NihalJain · 2025-05-27T10:21:27Z

Retriggered build!

…ation mechanism (Web UI only)" to branch-2 Co-authored-by: Yash Dodeja <[email protected]>

… class SchemaLdifExtractor in ApacheDS 2.0.0.AM26. The issue arises from an overridden ApacheDS version incompatible with Hadoop 2. To resolve, skip LDAP tests for Hadoop 2 and apply dependency changes only for Hadoop 3.

NihalJain · 2025-05-27T12:01:31Z

Rebased PR!

NihalJain · 2025-05-27T18:26:59Z

Failures are not related!

NihalJain · 2025-05-29T19:25:41Z

Failures are not related!

Apache-HBase · 2025-05-30T17:30:12Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 37s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ branch-2 Compile Tests _
+0 🆗	mvndep	0m 20s		Maven dependency ordering for branch
+1 💚	mvninstall	4m 45s		branch-2 passed
+1 💚	compile	7m 54s		branch-2 passed
+1 💚	checkstyle	1m 23s		branch-2 passed
+1 💚	spotbugs	10m 53s		branch-2 passed
+1 💚	spotless	1m 14s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 24s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 49s		the patch passed
+1 💚	compile	8m 35s		the patch passed
-0 ⚠️	javac	8m 35s	/results-compile-javac-root.txt	root generated 2 new + 1255 unchanged - 0 fixed = 1257 total (was 1255)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 31s	/results-checkstyle-root.txt	root: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	xmllint	0m 0s		No new issues.
+1 💚	spotbugs	9m 59s		the patch passed
+1 💚	hadoopcheck	23m 48s		Patch does not cause any errors with Hadoop 2.10.2 or 3.3.6 3.4.0.
+1 💚	spotless	0m 52s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 24s		The patch does not generate ASF License warnings.
		79m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5680/11/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5680
Optional Tests	dupname asflicense javac codespell detsecrets xmllint hadoopcheck spotless compile spotbugs checkstyle hbaseanti
uname	Linux 992f611804fc 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2 / `a644bb0`
Default Java	Eclipse Adoptium-11.0.23+9
Max. process+thread count	178 (vs. ulimit of 30000)
modules	C: hbase-http . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5680/11/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3 xmllint=20913
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-05-30T22:44:01Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
-0 ⚠️	yetus	0m 6s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ branch-2 Compile Tests _
+0 🆗	mvndep	0m 56s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 39s		branch-2 passed
+1 💚	compile	2m 16s		branch-2 passed
+1 💚	javadoc	2m 1s		branch-2 passed
+1 💚	shadedjars	6m 38s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 18s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 33s		the patch passed
+1 💚	compile	2m 30s		the patch passed
+1 💚	javac	2m 30s		the patch passed
+1 💚	javadoc	3m 2s		the patch passed
+1 💚	shadedjars	7m 21s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	275m 35s		root in the patch passed.
		315m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5680/11/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#5680
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 6e3bc9232607 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2 / `a644bb0`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5680/11/testReport/
Max. process+thread count	8535 (vs. ulimit of 30000)
modules	C: hbase-http . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5680/11/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-05-30T22:44:51Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 47s		Docker mode activated.
-0 ⚠️	yetus	0m 5s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ branch-2 Compile Tests _
+0 🆗	mvndep	0m 16s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 20s		branch-2 passed
+1 💚	compile	2m 17s		branch-2 passed
+1 💚	javadoc	2m 0s		branch-2 passed
+1 💚	shadedjars	6m 33s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 18s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 14s		the patch passed
+1 💚	compile	2m 14s		the patch passed
+1 💚	javac	2m 14s		the patch passed
+1 💚	javadoc	1m 57s		the patch passed
+1 💚	shadedjars	6m 36s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	294m 51s		root in the patch passed.
		331m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5680/11/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5680
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 49db72c2c452 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2 / `a644bb0`
Default Java	Eclipse Adoptium-11.0.23+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5680/11/testReport/
Max. process+thread count	8437 (vs. ulimit of 30000)
modules	C: hbase-http . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5680/11/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-05-31T00:47:25Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
-0 ⚠️	yetus	0m 4s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ branch-2 Compile Tests _
+0 🆗	mvndep	0m 42s		Maven dependency ordering for branch
+1 💚	mvninstall	2m 49s		branch-2 passed
+1 💚	compile	1m 48s		branch-2 passed
+1 💚	javadoc	1m 46s		branch-2 passed
+1 💚	shadedjars	5m 32s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 16s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 27s		the patch passed
+1 💚	compile	1m 49s		the patch passed
+1 💚	javac	1m 49s		the patch passed
+1 💚	javadoc	1m 46s		the patch passed
+1 💚	shadedjars	5m 32s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	423m 53s		root in the patch passed.
		455m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5680/11/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR	#5680
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 737fd4f5168c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2 / `a644bb0`
Default Java	Temurin-1.8.0_412-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5680/11/testReport/
Max. process+thread count	8112 (vs. ulimit of 30000)
modules	C: hbase-http . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5680/11/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

NihalJain · 2025-05-31T05:14:02Z

Green build !

…ation mechanism (Web UI only)" to branch-2 (apache#5144) (apache#5680) * Fix failure in pre-commit checks for Yetus JDK8 Hadoop2 due to missing class SchemaLdifExtractor in ApacheDS 2.0.0.AM26. The issue arises from an overridden ApacheDS version incompatible with Hadoop 2. To resolve, skip LDAP tests for Hadoop 2 and apply dependency changes only for Hadoop 3. Co-authored-by: Yash Dodeja <[email protected]> Signed-off-by: Nick Dimiduk <[email protected]>