Sequence-number-based replica allocation #46318
Description
When allocating a replica we prefer to allocate it on a node that already has a copy of the shard that is as close as possible to the primary, so that it is as cheap as possible to bring the new replica in sync with the primary. Indeed if we find a copy that is identical to the primary then we cancel an ongoing recovery on the grounds that a copy which is identical to the primary needs no work to recover as a replica.
We determine "as close as possible" by comparing the files on disk between the primary and replica, and "identical" by comparing the sync_id markers that were added by a synced flush. These mechanisms date back to before the introduction of sequence numbers, and do not always result in the best replica allocations in the presence of sequence-number-based recoveries. For instance, if two shard copies were allocated when the index was created then we do not expect them to have any segments in common; if additionally the copies were not synced-flushed then the ReplicaShardAllocator will consider them as completely different even though they might only differ in a small number of operations and be very cheap to recover.
We can improve this by making the ReplicaShardAllocator sensitive to sequence numbers. In many cases we maintain a peer-recovery retention lease (#41536) for copies of a shard that could reasonably be recovered by copying missing operations, so we can use the existence of such a lease to decide that a shard copy will be cheap to recover. In other cases where a shard is read-only (frozen, closed, ...) we can use sequence number information to determine that two copies are identical.