Expose all permits acquisition in IndexShard and TransportReplicationAction

[tlrx]

This pull request exposes two new methods in the IndexShard and TransportReplicationAction classes in order to allow transport replication actions to acquire all index shard operation permits for their execution.

It first adds the acquirePrimaryAllOperationsPermits() and the acquireReplicaAllOperationsPermits() methods to the IndexShard class which allow to acquire all operations permits on a shard while exposing a Releasable. It also refactors the TransportReplicationAction class to expose two protected methods (acquirePrimaryOperationPermit() and acquireReplicaOperationPermit()) that can be overridden when a transport replication action requires the acquisition of all permits on primary and/or replica shard during execution.

Finally, it adds a TransportReplicationAllPermitsAcquisitionTests which illustrates how a transport replication action can grab all permits before adding a cluster block in the cluster state, making subsequent operations that requires a single permit to fail (such test has been discussed in #35332 (comment)).

Related to elastic #33888

[elasticmachine]

Pinging @elastic/es-distributed

[bleskes]

Thx @tlrx. I left initial feedback on the production call.

[bleskes]

why is the targetAllocaitonID passed here? I guess it should be validated, but it isn't?

[tlrx]

This is a leftover that was helpful in TransportReplicationAllPermitsAcquisitionTests to dinstinguish between the primary and replica shard when executing a transport replication test action.

I reverted this and changed how the action in test retrieves the index shard.

[bleskes]

nit: acquireAllPrimaryOperationPermits?

[tlrx]

Done

[bleskes]

I'm not so comfortable with separating this code from the one in updatePrimaryTermIfNeeded - they are tightly connected. Instead of sharing code this way, how about creating a callback that will run:

indexShardOperationPermits.acquire(listener, executorOnDelay, true, debugInfo);

or

indexShardOperationPermits.asyncBlockOperations(listener, timeout.duration(), timeout.timeUnit());

[tlrx]

I share your concerns and I updated the code, let me know what you think.

[tlrx]

Thanks @bleskes, I addressed your first bunch of comments. I also added comments to the TransportReplicationAllPermitsAcquisitionTests and reworked it a bit so that it is easier to understand.

[bleskes]

LGTM. I left one suggestion for a potential follow up PR.

[bleskes]

This is indicates a funky property of getActiveOperationsCount. To date all the blocks were internal to the shard and thus 0 was a good return. I don't really have a good solution in mind. At the very least we should document this on getActiveOperationsCount. Maybe 1 (we know there is only one exclusive op) is a better return value? /cc @ywelsch

[tlrx]

I take the point, thanks.

[ywelsch]

sorry, I missed the ping here. I think that 1 is ok too, but then we should in general change the terminology in IndexShardOperationPermits to move from the notion of blocking operations to running exclusive operations (similar to a read/write lock).

[bleskes]

agreed.

[bleskes]

🙈

[tlrx]

I know, not really proud of this... I'll take a look at how to improve this test.