Remove node-level canAllocate override

[DaveCTurner]

Today there is a node-level canAllocate override which the balancer
uses to ignore certain nodes to which it is certain no more shards can
be allocated. In fact this override only ignores nodes which have hit
the rarely-used cluster.routing.allocation.total_shards_per_node
limit, so this optimization doesn't have a meaningful impact on real
clusters.

This commit removes this unnecessary fast path from the balancer, and
also removes all the machinery needed to support it.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Allocation)

[DaveCTurner]

@zuketo do we have any data that can show how rarely cluster.routing.allocation.total_shards_per_node is actually used? I couldn't see an obvious way to determine that from any telemetry.

[zuketo]

I couldn't find any good data sources for this (other than adding to telemetry). This setting is also not whitelisted by cloud, so no data points there. Could we deprecate first and then look at removal?

[DaveCTurner]

Thanks for confirming, Jason. TBC we're not talking about removing the setting itself, only the 100 lines of code that treats this setting as a special case in the shard allocator. Let's see what the team discussion brings.

[DaveCTurner]

Absent a better source of data I looked through as many user interactions as I could find and only encountered 7 mentions of this setting in the last 90 days, and I don't think any of them would have benefitted from this optimisation. We discussed this today as well and agreed to proceed.

[ywelsch]

LGTM