Expose allocation balance weights via endpoint

[idegtiarenko]

Description

BalancedShardAllocator uses a weight function while trying to evenly distribute the shard count, ingest load and disk usage across the nodes.

We should add an (internal?) endpoint to expose the weights and their sum across indices and nodes to be able to troubleshoot unbalanced clusters and understand what components are contributing to the lack of the overall balance.

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[billfischermnit]

Would it be possible/feasible to include the weight and the component parts (shard, index, disk usage, and write load) as an optional part of the _cat/nodes API response as well?

[idegtiarenko]

@billfischermnit, most of what you have listed is already available at /_cat/allocation.

[billfischermnit]

In 8.14, I see what I think would be 3 of the 4 components:

• shards
• write_load.forecast (but no current write_load ... does the weight function use current or forecast?)
• disk.used (is this the same value used by the weight function?)

... but it doesn't include the shards per index component, nor does it include the weights as calculated by the weight function. To understand the cluster from a desired balance perspective, one would need to take those _cat/allocation results and then calculate the weight based on that, which I'd argue would be impossible if one were just following the documentation, it could be error prone if one were reverse engineering the actual weight function code, and of course if the weight function changes, the off-line calculations would be incorrect too.

As such, if possible, I'd like to request that the weights as calculated by the weight function be exposed via the _cat/nodes and/or _cat/allocation APIs. To be fair, my preference would be for _cat/nodes just because I use that all the time,