HBASE-27951 Use ADMIN_QOS in MasterRpcServices for regionserver operational dependencies #5309

apurtell · 2023-06-23T23:02:48Z

It seems not correct to have the MasterRpcServices methods for direct regionserver<->master communication (regionServerStartup, regionServerReport, and reportFatalRSError) contending with normal priority requests. They should be made ADMIN_QOS priority to avoid potential operational deadlocks.

…tional dependencies It seems not correct to have the MasterRpcServices methods for direct regionserver<->master communication (regionServerStartup, regionServerReport, and reportFatalRSError) contending with normal priority requests. They should be made ADMIN_QOS priority to avoid potential operational deadlocks.

virajjasani

+1, based on some recent issues that we have seen

Apache-HBase · 2023-06-23T23:47:07Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 46s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 51s	master passed
+1 💚	compile	2m 29s	master passed
+1 💚	checkstyle	0m 37s	master passed
+1 💚	spotless	0m 43s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 29s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 32s	the patch passed
+1 💚	compile	2m 25s	the patch passed
+1 💚	javac	2m 25s	the patch passed
+1 💚	checkstyle	0m 33s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	9m 2s	Patch does not cause any errors with Hadoop 3.2.4 3.3.5.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 35s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 11s	The patch does not generate ASF License warnings.
		31m 39s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5309/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5309
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 99d469be4f18 5.4.0-148-generic #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `66fd6db`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	79 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5309/1/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-06-24T03:25:15Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	5m 30s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 29s	master passed
+1 💚	compile	0m 39s	master passed
+1 💚	shadedjars	4m 29s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 16s	the patch passed
+1 💚	compile	0m 39s	the patch passed
+1 💚	javac	0m 39s	the patch passed
+1 💚	shadedjars	4m 29s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	the patch passed
		_ Other Tests _
+1 💚	unit	224m 3s	hbase-server in the patch passed.
		249m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5309/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5309
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 160159f661a3 5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `66fd6db`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5309/1/testReport/
Max. process+thread count	4508 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5309/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-06-24T04:02:49Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 26s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 0s	master passed
+1 💚	compile	0m 46s	master passed
+1 💚	shadedjars	5m 13s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 46s	the patch passed
+1 💚	compile	0m 45s	the patch passed
+1 💚	javac	0m 45s	the patch passed
+1 💚	shadedjars	5m 10s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
-1 ❌	unit	264m 21s	hbase-server in the patch failed.
		287m 20s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5309/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5309
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux c8370b968e93 5.4.0-1099-aws #107~18.04.1-Ubuntu SMP Fri Mar 17 16:49:05 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `66fd6db`
Default Java	Eclipse Adoptium-11.0.17+8
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5309/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5309/1/testReport/
Max. process+thread count	4633 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5309/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

apurtell · 2023-06-26T16:05:33Z

Let me check locally that this failure is not related...

[ERROR] org.apache.hadoop.hbase.replication.TestNamespaceReplication.null
[ERROR]   Run 1: TestNamespaceReplication.testNamespaceReplication:165 Â» TestTimedOut test timed out after 1560 seconds
[ERROR]   Run 2: TestNamespaceReplication Â»  Appears to be stuck in thread MiniHBaseClusterRegionServer-EventLoopGroup-6-1

[ERROR] org.apache.hadoop.hbase.replication.TestNamespaceReplication.testNamespaceReplication[0: serialPeer=true]
[ERROR]   Run 1: TestNamespaceReplication.testNamespaceReplication:165 Â» TimeoutIO java.util.concurrent.TimeoutException
[ERROR]   Run 2: TestNamespaceReplication>TestReplicationBase.tearDownBase:315->TestReplicationBase.removePeer:309 Â» TimeoutIO java.util.concurrent.TimeoutException

Apache-HBase · 2023-06-26T23:55:49Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 58s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 30s	master passed
+1 💚	compile	2m 53s	master passed
+1 💚	checkstyle	0m 43s	master passed
+1 💚	spotless	0m 52s	branch has no errors when running spotless:check.
+1 💚	spotbugs	2m 5s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 25s	the patch passed
+1 💚	compile	3m 8s	the patch passed
+1 💚	javac	3m 8s	the patch passed
+1 💚	checkstyle	0m 43s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	9m 28s	Patch does not cause any errors with Hadoop 3.2.4 3.3.5.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 33s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 12s	The patch does not generate ASF License warnings.
		36m 12s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5309/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5309
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 7865a9a7c8a9 5.4.0-148-generic #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `66fd6db`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	80 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5309/2/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-06-27T03:17:53Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 20s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 51s	master passed
+1 💚	compile	0m 48s	master passed
+1 💚	shadedjars	4m 36s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 27s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 38s	the patch passed
+1 💚	compile	0m 47s	the patch passed
+1 💚	javac	0m 47s	the patch passed
+1 💚	shadedjars	4m 42s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	the patch passed
		_ Other Tests _
+1 💚	unit	216m 10s	hbase-server in the patch passed.
		238m 22s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5309/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5309
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 43181e374439 5.4.0-148-generic #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `66fd6db`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5309/2/testReport/
Max. process+thread count	4755 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5309/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-06-27T03:24:36Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 51s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 19s	master passed
+1 💚	compile	0m 39s	master passed
+1 💚	shadedjars	4m 29s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 18s	the patch passed
+1 💚	compile	0m 40s	the patch passed
+1 💚	javac	0m 40s	the patch passed
+1 💚	shadedjars	4m 27s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	the patch passed
		_ Other Tests _
+1 💚	unit	224m 0s	hbase-server in the patch passed.
		245m 4s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5309/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5309
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 348068c300cc 5.4.0-152-generic #169-Ubuntu SMP Tue Jun 6 22:23:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `66fd6db`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5309/2/testReport/
Max. process+thread count	4431 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5309/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…tional dependencies (#5309) It seems not correct to have the MasterRpcServices methods for direct regionserver<->master communication (regionServerStartup, regionServerReport, and reportFatalRSError) contending with normal priority requests. They should be made ADMIN_QOS priority to avoid potential operational deadlocks. Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: Viraj Jasani <[email protected]> Reviewed-by: Aman Poonia <[email protected]>

…regionserver operational dependencies (apache#5309) It seems not correct to have the MasterRpcServices methods for direct regionserver<->master communication (regionServerStartup, regionServerReport, and reportFatalRSError) contending with normal priority requests. They should be made ADMIN_QOS priority to avoid potential operational deadlocks. Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: Viraj Jasani <[email protected]> Reviewed-by: Aman Poonia <[email protected]>

…tional dependencies (apache#5309) It seems not correct to have the MasterRpcServices methods for direct regionserver<->master communication (regionServerStartup, regionServerReport, and reportFatalRSError) contending with normal priority requests. They should be made ADMIN_QOS priority to avoid potential operational deadlocks. Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: Viraj Jasani <[email protected]> Reviewed-by: Aman Poonia <[email protected]> (cherry picked from commit 37d62aa) Change-Id: I26ac371a2c2b2330d88d13b10aef21e0865d09ab

apurtell requested review from Apache9 and virajjasani June 23, 2023 23:03

apurtell force-pushed the HBASE-27951 branch from 2e68677 to 1036e61 Compare June 23, 2023 23:03

virajjasani approved these changes Jun 23, 2023

View reviewed changes

Apache9 approved these changes Jun 27, 2023

View reviewed changes

mnpoonia approved these changes Jun 28, 2023

View reviewed changes

apurtell merged commit 9e8e438 into apache:master Jun 28, 2023

apurtell deleted the HBASE-27951 branch June 28, 2023 19:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

HBASE-27951 Use ADMIN_QOS in MasterRpcServices for regionserver operational dependencies #5309

HBASE-27951 Use ADMIN_QOS in MasterRpcServices for regionserver operational dependencies #5309

Uh oh!

apurtell commented Jun 23, 2023

Uh oh!

virajjasani left a comment

Uh oh!

Apache-HBase commented Jun 23, 2023

Uh oh!

Apache-HBase commented Jun 24, 2023

Uh oh!

Apache-HBase commented Jun 24, 2023

Uh oh!

apurtell commented Jun 26, 2023

Uh oh!

Apache-HBase commented Jun 26, 2023

Uh oh!

Apache-HBase commented Jun 27, 2023

Uh oh!

Apache-HBase commented Jun 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

HBASE-27951 Use ADMIN_QOS in MasterRpcServices for regionserver operational dependencies #5309

HBASE-27951 Use ADMIN_QOS in MasterRpcServices for regionserver operational dependencies #5309

Uh oh!

Conversation

apurtell commented Jun 23, 2023

Uh oh!

virajjasani left a comment

Choose a reason for hiding this comment

Uh oh!

Apache-HBase commented Jun 23, 2023

Uh oh!

Apache-HBase commented Jun 24, 2023

Uh oh!

Apache-HBase commented Jun 24, 2023

Uh oh!

apurtell commented Jun 26, 2023

Uh oh!

Apache-HBase commented Jun 26, 2023

Uh oh!

Apache-HBase commented Jun 27, 2023

Uh oh!

Apache-HBase commented Jun 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants