A primary failing a replica should not count against the shard failures counter

[bleskes]

To avoid endless assignment loops we currently limit the number of times a shard will be allocated after failure. We only count real failures for these and ignore things like nodes dropping of the cluster. However, if the index is actively being indexed while a node is disconnected, the primary will request the master to fail the shard so it can ack the indexing operation. If that shard failure request reaches the master before the master process the node leaving the cluster, we increment the shard failure counter. If this happens repeatedly the shard will no longer be assigned and tests time out.

This is the cause of the failure of https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+dockeralpine-periodic/167/console

Since we designed the shard failure counter to protect against broken allocations (missing synonyms files etc.), we shouldn't count failures coming from the primary. This does come with the down side that we are not protected against partial network disconnects that the master doesn't see - these may then cause a replica to be allocated again and again.

[ywelsch]

Since we designed the shard failure counter to protect against broken allocations (missing synonyms files etc.), we shouldn't count failures coming from the primary. This does come with the down side that we are not protected against partial network disconnects that the master doesn't see - these may then cause a replica to be allocated again and again.

The underlying issue is to distinguish between failures that are temporary and those that are permanent and require user interaction to fix - such as the synonym files. If the network is flaky between primary and replica, would we prefer to fail the shard max_retries times and then leave it unassigned until user interaction? If we consider flaky networks as a temporary thing, it should not require user intervention to allocate the replica afterwards. This downside could be an upside.

[elasticmachine]

Pinging @elastic/es-distributed

[ywelsch]

@bleskes The situation that you described (repeatedly failing the replica when node with replica disconnects) should not happen anymore since we now only replicate writes to fully initialized shards, which means that the connection between primary and replica must first work before we start replicating changes between the two. I'm therefore closing this issue. Please let me know if you disagree with this assessment.

[bleskes]

+1. Thanks @ywelsch . Once a shard reaches the phase we replicated into it, we rely on the replication to work, so we should treat any failure as normal.