Exponential backoff of failed allocation

[clintongormley]

In #18467 we solved the problem where the failed allocation of a shard is retried in a tight loop, filling up the log file with exceptions. Now, after five failures, the allocation is no longer attempted until the user triggers it.

The downside of this approach is that is requires user intervention.

Would it be possible to add some kind of exponential backoff so that allocation attempts continue to be made, but with less frequency. That way we still avoid flooding the logs but if the situation resolves itself, the shard will be allocated automatically.

[clintongormley]

CC @ywelsch, @bleskes

[bleskes]

I think it's good to explore this. We can still keep the hard limit (and may increase it) - we built the feature for configuration mistakes - but delay the speed of re-assignment.

@clintongormley did you run into a specific issue that triggered this?

[clintongormley]

@bleskes Just from user feedback