Hard limit on total number of shards in a cluster

[alexbrasetvik]

We're seeing hundreds of cases of too many shards causing problems vs. problems caused by having too few.

It would be great to have a default hard limit, even though it can be increased (through the cluster settings API). It'll raise awareness to this issue hopefully in a "I can bump this now, but need to fix it"-way.

[javanna]

@alexbrasetvik I think this was done yesterday on #20682 ;)

[jasontedor]

@javanna That's different, that's per index but the request here is per cluster.

[alexbrasetvik]

Just synced up with @s1monw who asked me to create this issue while we talked about the per-index limit. This one is indeed per cluster, as a total number of shards - whether it's a few indices with a lot of shards, or many single-shard indices.

[javanna]

sounds good thanks for clarifying.

[clintongormley]

max_shards_per_node

This setting would be checked on user actions like create index, restore snapshot, open index. If the total number of shards in the cluster is greater than max_shards_per_node * number_of_nodes then the user action can be rejected. This implementation allows the max value to be exceeded if (eg) a node fails, resulting in a lower total max shards per cluster.

We would default to a high number during 5.x (eg 1000), giving sysadmins the ability to set it to whatever makes sense for their cluster, and we can look at lowering this value for 6.0.

[gmoskovicz]

We would default to a high number during 5.x (eg 1000), giving sysadmins the ability to set it to whatever makes sense for their cluster, and we can look at lowering this value for 6.0.

I would say that ~500/600 shards per node is a good limit.