Disk-based shard allocation does not take into account freshly-relocating shards

[DaveCTurner]

Disk-based shard allocation may decide to allocate a shard to a node as long as that node has enough free space for that shard on its emptiest data path, accounting for the sizes of shards that are already relocating to that data path on that node. However, it does not currently account for shards that have been chosen for relocation to a node but whose assignment to a data path has not yet been observed by the ClusterInfoService. In particular, it ignores relocations triggered earlier on in the same reroute process, because these relocations have not been published to the cluster so their effects cannot have been observed.

This is particularly problematic if cluster.routing.allocation.node_concurrent_recoveries is set to a high value (e.g. 10): if the shards are large enough then this may cause the target node to breach one or more watermarks, possibly requiring further allocations away from the node to fix.

[elasticmachine]

Pinging @elastic/es-distributed

[DaveCTurner]

Additionally, disk-based shard allocation does not correctly take into account the sizes of shards that were previously changed to RELOCATING during the same reroute process, i.e. those shards that have just started moving off a node. The issue here is that we look up the path of a shard based on its ShardRouting, but if we just started to relocate the shard then we just changed its ShardRouting (typically from a STARTED to a RELOCATING one) so the path lookup is lost and the relocating shard is ignored. The effect is that a node that exceeds its high watermark may attempt to relocate every single one of its shards away, until some other limit (e.g. the ThrottlingAllocationDecider) takes effect.