A little supplement about ThrottlingAllocationDecider

[kkewwei]

ES_VERSION: 7.3.1
JVM version : JDK1.8.0_112
OS version : linux
Description of the problem including expected versus actual behavior:
       The shard balancer uses ThrottlingAllocationDecider to throttle the concurrency of allocating, and use node_concurrent_incoming_recoveries and node_concurrent_outgoing_recoveries to throttle the concurrency of each node, it should be good.
       In product, I migrate data from several nodes(such as 20 nodes) to another several nodes(such as 20 nodes) in one cluster. when I set node_concurrent_incoming_recoveries and node_concurrent_outgoing_recoveries to be 1, the concurrency of moving is 20, which raises a serious question: writing performance of clusters is greatly affected. the concurrency of 20 is too high. I has to open the moving at night, and close moving during the day time.
       If only I could control the concurrency at cluster level, not just a node, In this case, cluster_concurrent_rebalance doesn't work. Can we add a cluster rebalance/moving concurrency here no matter it is moving or rebalance, allocating?

[elasticmachine]

Pinging @elastic/es-distributed

[DaveCTurner]

@kkewwei it is surprising to me that your cluster cannot tolerate one inbound or outbound recovery from each node. What is the bottleneck? If it's network bandwidth, you can use indices.recovery.max_bytes_per_sec to control that.

[kkewwei]

@DaveCTurner I find that indices.recovery.max_bytes_per_sec is set a bit high, this is the reason. do you think should we set a global concurrency about moving, rebalance, allocating?

[DaveCTurner]

@kkewwei Do you mean that the default for indices.recovery.max_bytes_per_sec (which is 40MB/s) is too high? Or do you mean that you adjusted it?

I don't think a global limit on the number of moving shards is a good idea. We can expect larger clusters to have relatively complex network topologies, meaning that some collections of shard movements will run smoothly while others of the same size might hit a bottleneck. I don't have a good alternative to suggest, however, so I'm opening this for further discussion.

[kkewwei]

I adjusted it. On one hand I set the node_concurrent_incoming_recoveries and node_concurrent_outgoing_recoveries to be 50 or bigger to recovery faster when a node leaves the cluster and rejoins the cluster, it is very common in product. On other hand, when I exclude some nodes using cluster.routing.allocation.exclude._ip, there are hundreds of shards moving to other node at the moment, if we could set global limit on the number of moving shards. This is my experience, which may be not good suggest.

[DaveCTurner]

I set the node_concurrent_incoming_recoveries and node_concurrent_outgoing_recoveries to be 50 or bigger

I am confused, your original post was about performance when these are set to 1. I'd definitely expect performance issues if you set these to 50 or larger. I recommend leaving those settings at their defaults.

[DaveCTurner]

@kkewwei we're still waiting for clarification here. Do you see issues with node_concurrent_recoveries and indices.recovery.max_bytes_per_sec at their default values, or does it only cause you problems when you increase them?

[kkewwei]

sorry. I increase those two settings to be higher and cause the problem. the default value of node_concurrent_recoveries is not suitable for our cluster, because there have thousands of shards on every node in the cluster, It is unacceptable to speed too much time recovery with the default.

[DaveCTurner]

Thanks for the clarification. I think the solution for you is not to set these settings so high. They are a safety mechanism to prevent recoveries from overloading your cluster. If you disable safety mechanisms like this then I think overloading your cluster is expected.

You should not have thousands of shards on every node in the cluster. In 7.3.1 you are prevented from doing this via the cluster.max_shards_per_node setting, which is another safety mechanism that you must have disabled. I don't think it's surprising that you might see performance issues in such a configuration.

You may see an improvement in recovery performance in 7.4.0 thanks to #44433. You may also want to see if an increase indices.recovery.max_concurrent_file_chunks improves performance.

If you can help us identify an issue in a cluster with its safety mechanisms in place then please do so, but for now I am closing this issue for the reasons described above.