HBASE-28109 NPE for the region state: Failed to become active master (HMaster) #5432

hanke580 · 2023-09-23T21:01:19Z

There's a chance for rs to be NULL when starting up the HMaster. Then we ran into NPE exception

2023-09-18 14:17:40,452 ERROR [master/hmaster:16000:becomeActiveMaster] master.HMaster: Failed to become active master
java.lang.NullPointerException
        at org.apache.hadoop.hbase.master.HMaster.isRegionOnline(HMaster.java:1229)
        at org.apache.hadoop.hbase.master.HMaster.waitForMetaOnline(HMaster.java:1218)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:968)
        at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193)
        at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528)
        at java.lang.Thread.run(Thread.java:750)

This PR add a simple check and make HMaster keep waiting instead of directly shutting down.

Apache-HBase · 2023-09-23T21:47:04Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 32s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 48s	master passed
+1 💚	compile	2m 27s	master passed
+1 💚	checkstyle	0m 38s	master passed
+1 💚	spotless	0m 44s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 28s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 35s	the patch passed
+1 💚	compile	2m 24s	the patch passed
+1 💚	javac	2m 24s	the patch passed
+1 💚	checkstyle	0m 35s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	9m 9s	Patch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 36s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 12s	The patch does not generate ASF License warnings.
		31m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5432/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5432
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 38bccac8e995 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `05f5d5b`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	79 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5432/1/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-09-24T01:13:05Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 32s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 12s	master passed
+1 💚	compile	0m 47s	master passed
+1 💚	shadedjars	5m 27s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 47s	the patch passed
+1 💚	compile	0m 44s	the patch passed
+1 💚	javac	0m 44s	the patch passed
+1 💚	shadedjars	5m 22s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	the patch passed
		_ Other Tests _
+1 💚	unit	212m 33s	hbase-server in the patch passed.
		236m 3s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5432/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5432
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux b6e5fcaee43d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `05f5d5b`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5432/1/testReport/
Max. process+thread count	4686 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5432/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-09-24T01:16:40Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 26s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 36s	master passed
+1 💚	compile	0m 37s	master passed
+1 💚	shadedjars	5m 12s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 11s	the patch passed
+1 💚	compile	0m 36s	the patch passed
+1 💚	javac	0m 36s	the patch passed
+1 💚	shadedjars	5m 11s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 21s	the patch passed
		_ Other Tests _
-1 ❌	unit	219m 28s	hbase-server in the patch failed.
		241m 13s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5432/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5432
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 763fbbf4ca85 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `05f5d5b`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5432/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5432/1/testReport/
Max. process+thread count	4672 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5432/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2023-09-24T08:10:13Z

Is this a fresh new cluster? Otherwise I do not think the region state for meta could be null...

hanke580 · 2023-09-24T15:08:04Z

@Apache9 Yes, it a fresh cluster start up.

…(HMaster) (#5432) The RegionState for meta region could be null for a fresh new cluster, we should also wait a bit to let it finish the initialization. Signed-off-by: Duo Zhang <[email protected]> (cherry picked from commit 865a595)

…(HMaster) (apache#5432) The RegionState for meta region could be null for a fresh new cluster, we should also wait a bit to let it finish the initialization. Signed-off-by: Duo Zhang <[email protected]>

…(HMaster) (apache#5432) The RegionState for meta region could be null for a fresh new cluster, we should also wait a bit to let it finish the initialization. Signed-off-by: Duo Zhang <[email protected]> (cherry picked from commit 865a595) (cherry picked from commit 736595f) Change-Id: Ic14da19443721859d5a403cd41d282dc58a39867

Avoid NPE when starting up HMaster

e4ce180

Apache9 approved these changes Oct 6, 2023

View reviewed changes

Apache9 changed the title ~~HBASE-28109 Avoid NPE when starting up HMaster~~ HBASE-28109 NPE for the region state: Failed to become active master (HMaster) Oct 7, 2023

Apache9 merged commit 865a595 into apache:master Oct 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HBASE-28109 NPE for the region state: Failed to become active master (HMaster) #5432

HBASE-28109 NPE for the region state: Failed to become active master (HMaster) #5432

Uh oh!

hanke580 commented Sep 23, 2023

Uh oh!

Apache-HBase commented Sep 23, 2023

Uh oh!

Apache-HBase commented Sep 24, 2023

Uh oh!

Apache-HBase commented Sep 24, 2023

Uh oh!

Apache9 commented Sep 24, 2023

Uh oh!

hanke580 commented Sep 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HBASE-28109 NPE for the region state: Failed to become active master (HMaster) #5432

HBASE-28109 NPE for the region state: Failed to become active master (HMaster) #5432

Uh oh!

Conversation

hanke580 commented Sep 23, 2023

Uh oh!

Apache-HBase commented Sep 23, 2023

Uh oh!

Apache-HBase commented Sep 24, 2023

Uh oh!

Apache-HBase commented Sep 24, 2023

Uh oh!

Apache9 commented Sep 24, 2023

Uh oh!

hanke580 commented Sep 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants