Control Cluster Shards Balancing #42739
Conversation
matriv
added
the
:Distributed Coordination/Allocation
|
Pinging @elastic/es-distributed |
ywelsch
left a comment
ywelsch
left a comment
There was a problem hiding this comment.
Thank you for your interest in contributing to ES. I don't understand what this change is trying to achieve, unfortunately. Can you provide some more explanations? In particular, why is it removing the code that allows index-level settings from overriding the cluster-level ones?
|
If an index is allowed to be rebalance, then this line of code "if (allocation. deciders (). canRebalance (allocation). type ()!= Type. YES)" returns false, and the code continues to execute downward. In the balanceByWeights () method, all indexes are traversed according to the weight rebalance, so I set cluster. routing. rebalance. enable: "none" is invalid. |
ywelsch
left a comment
ywelsch
left a comment
There was a problem hiding this comment.
I'm not sure I understand what the purpose of this PR is, especially as it's disabling existing ES functionality. Is it to address a scaling issue? You mention a cluster with 300,000 shards on another PR, so I assume this is related. Why is this cluster making use of index-level rebalancing (i.e. the index.routing.rebalance.enable setting) if that is turning out to be a scaling bottleneck?
| return allocation.decision(Decision.NO, NAME, "none rebalance are not allowed"); | ||
| case PRIMARIES: | ||
| if (allocation.routingNodes().hasInactivePrimaries()) { | ||
| return allocation.decision(Decision.NO, NAME, |
There was a problem hiding this comment.
why does this globally disable rebalancing when there are some inactive primaries of an unrelated index?
| return allocation.decision(Decision.YES, NAME, "all primary shards is active and rebalance are allowed"); | ||
| case REPLICAS: | ||
| if (allocation.routingNodes().hasInactiveShards()) { | ||
| return allocation.decision(Decision.NO, NAME, |
There was a problem hiding this comment.
why does this globally disable rebalancing when there are inactive shards of some unrelated index?
| case ALL: | ||
| return allocation.decision(Decision.YES, NAME, "all rebalance are allowed"); | ||
| case NONE: | ||
| return allocation.decision(Decision.NO, NAME, "none rebalance are not allowed"); |
There was a problem hiding this comment.
As said earlier, this changes the behavior of ES not to take the index-level property into account anymore when the cluster-level property is set.
4 participants
Previous implementations were well thought out, but an index balance would cause all indexes to balance together.