Allocation awareness does not respect allocation excludes

[DaveCTurner]

Today if you are using allocation awareness and you wish to decommission an entire zone, you may not be able to gracefully vacate all shards from this zone since allocation awareness computes the maximum number of shards per zone without regard for any allocation filters, and these constraints may conflict.

We will address this with #49064, but we may like to consider having allocation awareness respect certain cluster-wide allocation exclusions too, as a special case to support zone-wide vacation. We saw one case where this might have been useful recently, so I'm opening this to collect other evidence that this would be useful.

Relates #8178, #49064.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Allocation)

[nik9000]

@DaveCTurner I think I remember you doing something on allocation awareness recently. Did that cover this?

[DaveCTurner]

Nope. Today allocation awareness plays more nicely with auto-expand replicas (#69334) and auto-expand replicas plays nicely with allocation filtering (#48974). This issue is about making allocation awareness play nicely with allocation filtering, and "plays nicely" is sadly not transitive.

[DaveCTurner]

A similar problem applies to data tiers, see e.g. this forum post. Data tiers aren't quite the same as allocation filtering, but they're close enough relations that they both suffer from this issue.