-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Open
Labels
:Distributed Coordination/AllocationAll issues relating to the decision making around placing a shard (both master logic & on the nodes)All issues relating to the decision making around placing a shard (both master logic & on the nodes)>enhancement>tech debtSupportabilityImprove our (devs, SREs, support eng, users) ability to troubleshoot/self-service product better.Improve our (devs, SREs, support eng, users) ability to troubleshoot/self-service product better.Team:Distributed (Obsolete)Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.
Description
Today IndexRoutingTable$Builder#build constructs an all-new IndexShardRoutingTable for each shard on every change. High-traffic clusters may have large numbers of shards per index, and today's behaviour yields many duplicate IndexShardRoutingTable instances which together may consume a substantial amount of heap on the elected master.
Moreover IndexRoutingTable is a SimpleDiffable so we must transmit the whole table over the wire on any change, which prevents sharing on the receiving node and therefore yields many duplicate IndexShardRoutingTable instances on the other nodes in the cluster.
We should:
- Re-use existing
IndexShardRoutingTableinstances where possible when adjusting the routing table on the elected master - Refine the diffability of the
IndexRoutingTableso that it avoids sending unchangedIndexShardRoutingTableinstances over the wire.
Relates #77466
Metadata
Metadata
Assignees
Labels
:Distributed Coordination/AllocationAll issues relating to the decision making around placing a shard (both master logic & on the nodes)All issues relating to the decision making around placing a shard (both master logic & on the nodes)>enhancement>tech debtSupportabilityImprove our (devs, SREs, support eng, users) ability to troubleshoot/self-service product better.Improve our (devs, SREs, support eng, users) ability to troubleshoot/self-service product better.Team:Distributed (Obsolete)Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.