Timeout for primary relocation handoff seems too long

[DaveCTurner]

Today we wait for 30 minutes to block all operations when performing a primary relocation handoff:

elasticsearch/server/src/main/java/org/elasticsearch/index/shard/IndexShard.java

Line 636 in 4bd8e7b

indexShardOperationPermits.blockOperations(30, TimeUnit.MINUTES, () -> {

However this behaves badly if a write task gets stuck, for instance in the case fixed in #36770. The effect is that the block cannot be put in place (because there is an in-flight operation) but the pending block prevents any further operations from taking place until it eventually fails.

I think we should consider reducing this timeout, because it seems preferable to fail the primary relocation (and therefore resume write operations) much sooner than 30 minutes.

[elasticmachine]

Pinging @elastic/es-distributed

[DaveCTurner]

We discussed this again. The stuck write task was a bug that is fixed. There's no good value for this timeout: for instance 60 seconds is too short (draining the remaining indexing tasks really can take that long) and also too long (you will definitely perceive a 60-second pause in indexing as an outage). Lacking agreement on a better timeout, and having not seen any further occurrences of this, let's close it.