HBASE-29544: Assertion errors in BackupAndRestoreThread are not causing IntegrationTestBackupRestore to fail #7252

kgeisz · 2025-08-27T17:16:01Z

Summary

https://issues.apache.org/jira/browse/HBASE-29544

This is a cherry-pick from PR #7243.

IntegrationTestBackupRestore runs the backup and restore processes in a separate thread (or threads). PR #7203 (HBASE-29503) fixed an issue where the test would still pass even if an exception occurred in one of the threads. However, that fix only handles exceptions. It does not handle other throwables, such as AssertionError. This pull request extends the functionality implemented in PR #7203 to handle all throwables instead of just exceptions.

Manual Testing of Errors

Exceptions

I added a RuntimeException to BackupAndRestoreThread.run() to test exception handling:

public void run() {
  try {
    # Added this to force an exception
    throw new RuntimeException("Do we see this?");
  } catch (Throwable t) {
    LOG.error(
      "An error occurred in thread {} when performing a backup and restore with table {}: ",
      Thread.currentThread().getName(), this.table.getNameAsString(), t);
    this.throwable = t;
  }
}

Error message (note the Caused by...):

java.lang.AssertionError: An error occurred in a backup and restore thread

	at org.apache.hadoop.hbase.IntegrationTestBackupRestore.runTestMulti(IntegrationTestBackupRestore.java:238)
	at org.apache.hadoop.hbase.IntegrationTestBackupRestore.testBackupRestore(IntegrationTestBackupRestore.java:207)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
Caused by: java.lang.RuntimeException: Do we see this?
	at org.apache.hadoop.hbase.IntegrationTestBackupRestore$BackupAndRestoreThread.run(IntegrationTestBackupRestore.java:127)
	at java.base/java.lang.Thread.run(Thread.java:840)

AssertionErrors

I forced an AssertionError by modifying the assertEquals() in restoreVerifyTable():

Assert.assertEquals(-1, util.countRows(hTable));

Error message (note the Caused by...):

java.lang.AssertionError: An error occurred in a backup and restore thread

	at org.apache.hadoop.hbase.IntegrationTestBackupRestore.runTestMulti(IntegrationTestBackupRestore.java:238)
	at org.apache.hadoop.hbase.IntegrationTestBackupRestore.testBackupRestore(IntegrationTestBackupRestore.java:207)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
Caused by: java.lang.AssertionError: expected:<-1> but was:<10000>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.hadoop.hbase.IntegrationTestBackupRestore.restoreVerifyTable(IntegrationTestBackupRestore.java:346)
	at org.apache.hadoop.hbase.IntegrationTestBackupRestore.runTestSingle(IntegrationTestBackupRestore.java:319)
	at org.apache.hadoop.hbase.IntegrationTestBackupRestore$BackupAndRestoreThread.run(IntegrationTestBackupRestore.java:127)
	at java.base/java.lang.Thread.run(Thread.java:840)

…ng IntegrationTestBackupRestore to fail (apache#7243) Signed-off-by: Tak Lon (Stephen) Wu <[email protected]>

Apache-HBase · 2025-08-27T17:51:10Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 45s		Docker mode activated.
-0 ⚠️	yetus	0m 6s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ branch-2.6 Compile Tests _
+1 💚	mvninstall	2m 39s		branch-2.6 passed
+1 💚	compile	0m 18s		branch-2.6 passed
+1 💚	javadoc	0m 13s		branch-2.6 passed
+1 💚	shadedjars	5m 39s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 30s		the patch passed
+1 💚	compile	0m 18s		the patch passed
+1 💚	javac	0m 18s		the patch passed
+1 💚	javadoc	0m 11s		the patch passed
+1 💚	shadedjars	5m 31s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	0m 55s		hbase-it in the patch passed.
		20m 8s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7252/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR	#7252
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 30dd2510eb00 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2.6 / `03f7476`
Default Java	Temurin-1.8.0_412-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7252/1/testReport/
Max. process+thread count	816 (vs. ulimit of 30000)
modules	C: hbase-it U: hbase-it
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7252/1/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-08-27T17:56:07Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 43s		Docker mode activated.
-0 ⚠️	yetus	0m 5s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ branch-2.6 Compile Tests _
+1 💚	mvninstall	3m 20s		branch-2.6 passed
+1 💚	compile	0m 21s		branch-2.6 passed
+1 💚	javadoc	0m 14s		branch-2.6 passed
+1 💚	shadedjars	6m 26s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 8s		the patch passed
+1 💚	compile	0m 21s		the patch passed
+1 💚	javac	0m 21s		the patch passed
+1 💚	javadoc	0m 12s		the patch passed
+1 💚	shadedjars	6m 26s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	0m 53s		hbase-it in the patch passed.
		23m 8s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7252/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7252
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 6a6494c6c972 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2.6 / `03f7476`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7252/1/testReport/
Max. process+thread count	850 (vs. ulimit of 30000)
modules	C: hbase-it U: hbase-it
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7252/1/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-08-27T18:05:51Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 45s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ branch-2.6 Compile Tests _
+1 💚	mvninstall	4m 3s		branch-2.6 passed
+1 💚	compile	0m 43s		branch-2.6 passed
+1 💚	checkstyle	0m 15s		branch-2.6 passed
+1 💚	spotbugs	0m 34s		branch-2.6 passed
+1 💚	spotless	1m 4s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 41s		the patch passed
+1 💚	compile	0m 32s		the patch passed
+1 💚	javac	0m 32s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 12s		the patch passed
+1 💚	spotbugs	0m 36s		the patch passed
+1 💚	hadoopcheck	19m 4s		Patch does not cause any errors with Hadoop 2.10.2 or 3.3.6 3.4.0.
+1 💚	spotless	1m 4s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 10s		The patch does not generate ASF License warnings.
		34m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7252/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7252
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux 2dd41e103680 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2.6 / `03f7476`
Default Java	Eclipse Adoptium-11.0.23+9
Max. process+thread count	77 (vs. ulimit of 30000)
modules	C: hbase-it U: hbase-it
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7252/1/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-08-27T18:06:56Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 10s		Docker mode activated.
-0 ⚠️	yetus	0m 6s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ branch-2.6 Compile Tests _
+1 💚	mvninstall	5m 21s		branch-2.6 passed
+1 💚	compile	0m 37s		branch-2.6 passed
+1 💚	javadoc	0m 28s		branch-2.6 passed
+1 💚	shadedjars	9m 21s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	5m 2s		the patch passed
+1 💚	compile	0m 34s		the patch passed
+1 💚	javac	0m 34s		the patch passed
+1 💚	javadoc	0m 18s		the patch passed
+1 💚	shadedjars	9m 36s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	1m 19s		hbase-it in the patch passed.
		35m 18s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7252/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#7252
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 87768359a9ff 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2.6 / `03f7476`
Default Java	Eclipse Adoptium-11.0.23+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7252/1/testReport/
Max. process+thread count	864 (vs. ulimit of 30000)
modules	C: hbase-it U: hbase-it
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7252/1/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

HBASE-29544: Assertion errors in BackupAndRestoreThread are not causi…

03f7476

…ng IntegrationTestBackupRestore to fail (apache#7243) Signed-off-by: Tak Lon (Stephen) Wu <[email protected]>

taklwu approved these changes Aug 27, 2025

View reviewed changes

taklwu merged commit 2baf739 into apache:branch-2.6 Aug 27, 2025
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

HBASE-29544: Assertion errors in BackupAndRestoreThread are not causing IntegrationTestBackupRestore to fail #7252

HBASE-29544: Assertion errors in BackupAndRestoreThread are not causing IntegrationTestBackupRestore to fail #7252

Uh oh!

kgeisz commented Aug 27, 2025

Uh oh!

Apache-HBase commented Aug 27, 2025

Uh oh!

Apache-HBase commented Aug 27, 2025

Uh oh!

Apache-HBase commented Aug 27, 2025

Uh oh!

Apache-HBase commented Aug 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

HBASE-29544: Assertion errors in BackupAndRestoreThread are not causing IntegrationTestBackupRestore to fail #7252

HBASE-29544: Assertion errors in BackupAndRestoreThread are not causing IntegrationTestBackupRestore to fail #7252

Uh oh!

Conversation

kgeisz commented Aug 27, 2025

Summary

Manual Testing of Errors

Exceptions

AssertionErrors

Uh oh!

Apache-HBase commented Aug 27, 2025

Uh oh!

Apache-HBase commented Aug 27, 2025

Uh oh!

Apache-HBase commented Aug 27, 2025

Uh oh!

Apache-HBase commented Aug 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants