HBASE-29071 StochasticLoadBalancer candidate generators should use a Map, rather than ordinal based indexing #6598

rmdmattingly · 2025-01-13T15:00:10Z

This just makes the candidate generators nicer to work with. This is part of #6593 that I'm breaking off into an atomic piece

Existing test coverage provides great guardrails to ensure that we haven't broken anything

rmdmattingly · 2025-01-14T14:48:48Z

...t/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancerBalanceCluster.java

  @Test
  public void testBalanceCluster() throws Exception {
-    setMaxRunTime(Duration.ofMillis(1500));
+    setMaxRunTime(Duration.ofMillis(2500));


I think I was a little ambitious with the duration required here (in #6597 I drastically reduced balancer test suite runtime), and this test is much less flexible because it doesn't follow the standard balancer test framework. I could bite off a larger refactor, but I'm not sure it's worth it. So, anyway, I saw some flappiness in this test locally that was resolved by just giving a little bit more time for plan generation since it expects the plan to be perfect in one shot

I think you'll find that the upstream execution needs quite a lot more time than local runs. Congratulations on your first flakey test!

...e-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java

ndimiduk · 2025-01-15T13:40:19Z

...e-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java

  void initCosts(BalancerClusterState cluster) {
    // Initialize the weights of generator every time
-    weightsOfGenerators = new double[this.candidateGenerators.size()];
+    weightsOfGenerators = new HashMap<>(this.candidateGenerators.size());


You're using getOrDefault everywhere, so can't you just clear() the map instead of re-seeding it with default values? Or you're protecting against non-default accesses from derived classes? In that case, make the instance private and force derived classes to interact with its contents though accessor methods.

Wait, it is private. Yeah, just clear it in the init function and access it via getOrDefault.

ndimiduk · 2025-01-15T13:45:15Z

...t/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancerBalanceCluster.java

  @Test
  public void testBalanceCluster() throws Exception {
-    setMaxRunTime(Duration.ofMillis(1500));
+    setMaxRunTime(Duration.ofMillis(2500));


I think you'll find that the upstream execution needs quite a lot more time than local runs. Congratulations on your first flakey test!

ndimiduk · 2025-01-15T13:46:59Z

...e-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java

+      partialSums.set(i, partialSums.get(i) / sum);
+    }
+
+    // Generate a random number and pick the first generator whose partial sum is >= rand


This weighted selection algorithm is pretty awkward to follow if you don't expect it. Can you isolate it within its own method so that the calling method is easier to read and the implementation could be picked up for reuse later. Also, it becomes testable independently, if such a thing is testable.

Agreed, I had some trouble understanding exactly what the goals were when working this the original code. I want to flag that this weighted selection algorithm is a pretty faithful attempt at maintaining the original approach — but with maps rather than arrays, some extra comments to try to explain the steps more explicitly, explicitly throwing rather than fetching a nonexistent item if generators are empty, falling back to random generators rather than some magic importance on the last in the list, etc

I think I can clean this up a bit with the precondition and maybe pulling random generator picking out into its own method — but the logic necessary for picking a random generator is basically the entirety of this. Unless we want to make a static method with a ton of arguments just for testing's sake, and honestly I think this is really well tested by the current suite; if you don't pick the right generators at the right time, then the balancer doesn't work well at all

ndimiduk · 2025-01-15T13:53:26Z

...e-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java

+      logMessage.append(String.format(" - %s: %d steps, %d approvals%n", generator.getSimpleName(),
+        count, approvals));
+    });
+    LOG.info(logMessage.toString());


Should this really be logged at INFO level? Seems more like DEBUG to me.

ndimiduk · 2025-01-15T13:54:35Z

...e-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java

+    StringBuilder logMessage = new StringBuilder("CandidateGenerator activity summary:\n");
+    generatorToStepCount.forEach((generator, count) -> {
+      long approvals = generatorToApprovedActionCount.getOrDefault(generator, 0L);
+      logMessage.append(String.format(" - %s: %d steps, %d approvals%n", generator.getSimpleName(),


use a String joiner here instead of naive append so that you don't have the extra newline at the end.

…Map, rather than ordinal based indexing

ndimiduk

Nice one @rmdmattingly

ndimiduk · 2025-01-21T12:45:57Z

I don't understand why this build is marked as failed -- the console output is all clear. https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6598/7/console

Apache-HBase · 2025-01-21T13:27:00Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 30s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+0 🆗	mvndep	0m 38s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 22s		master passed
+1 💚	compile	3m 31s		master passed
+1 💚	checkstyle	0m 45s		master passed
+1 💚	spotbugs	1m 55s		master passed
+1 💚	spotless	0m 44s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 52s		the patch passed
+1 💚	compile	3m 27s		the patch passed
+1 💚	javac	0m 22s		hbase-balancer generated 0 new + 57 unchanged - 13 fixed = 57 total (was 70)
+1 💚	javac	3m 5s		hbase-server in the patch passed.
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 44s		the patch passed
+1 💚	spotbugs	2m 8s		the patch passed
+1 💚	hadoopcheck	10m 57s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 42s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 18s		The patch does not generate ASF License warnings.
		39m 45s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6598/8/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6598
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux 124203dcfabd 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b8ab794`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	85 (vs. ulimit of 30000)
modules	C: hbase-balancer hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6598/8/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-01-21T17:29:43Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 42s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+0 🆗	mvndep	0m 44s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 57s		master passed
+1 💚	compile	1m 22s		master passed
+1 💚	javadoc	0m 47s		master passed
+1 💚	shadedjars	6m 4s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 12s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 28s		the patch passed
+1 💚	compile	1m 26s		the patch passed
+1 💚	javac	1m 26s		the patch passed
+1 💚	javadoc	0m 53s		the patch passed
+1 💚	shadedjars	6m 32s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	7m 16s		hbase-balancer in the patch passed.
-1 ❌	unit	243m 34s	/patch-unit-hbase-server.txt	hbase-server in the patch failed.
		282m 21s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6598/8/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6598
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 64ac34fe456e 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b8ab794`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6598/8/testReport/
Max. process+thread count	4445 (vs. ulimit of 30000)
modules	C: hbase-balancer hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6598/8/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

rmdmattingly · 2025-01-21T17:38:52Z

Test failures are disappointing, but irrelevant

…Map, rather than ordinal based indexing (#6598) Co-authored-by: Ray Mattingly <[email protected]>

…Map, rather than ordinal based indexing (#6598) (#6621) Co-authored-by: Ray Mattingly <[email protected]> Signed-off-by: Nick Dimiduk <[email protected]>

…Map, rather than ordinal based indexing (#6598) (#6620) Co-authored-by: Ray Mattingly <[email protected]> Signed-off-by: Nick Dimiduk <[email protected]>

…Map, rather than ordinal based indexing (#6598) (#6621) Co-authored-by: Ray Mattingly <[email protected]> Signed-off-by: Nick Dimiduk <[email protected]>

…Map, rather than ordinal based indexing (apache#6598) Co-authored-by: Ray Mattingly <[email protected]>

…Map, rather than ordinal based indexing (#6598) (#6621) Co-authored-by: Ray Mattingly <[email protected]> Signed-off-by: Nick Dimiduk <[email protected]>

…Map, rather than ordinal based indexing (#6598) (#6621) (#6631) Signed-off-by: Nick Dimiduk <[email protected]> Co-authored-by: Ray Mattingly <[email protected]>

…Map, rather than ordinal based indexing (apache#6598) (apache#6621) Co-authored-by: Ray Mattingly <[email protected]> Signed-off-by: Nick Dimiduk <[email protected]>

…tors should use a Map, rather than ordinal based indexing (apache#6598) (apache#6621) (will be in 2.6.3) Co-authored-by: Ray Mattingly <[email protected]> Signed-off-by: Nick Dimiduk <[email protected]>

…Map, rather than ordinal based indexing (apache#6598) (apache#6621) Co-authored-by: Ray Mattingly <[email protected]> Signed-off-by: Nick Dimiduk <[email protected]>

rmdmattingly mentioned this pull request Jan 13, 2025

HBASE-28513 The StochasticLoadBalancer should support discrete evaluations #6593

Closed