Allow adjustment of index resource constraints in ILM phase transitions

[DaveCTurner]

The shards allocator mostly does a good job of spreading the shards of each index out across the available nodes, but there are some corner cases where it might temporarily concentrate too many hot shards on a single node, leading to poor performance and possible resource exhaustion (#17213). It is occasionally useful to add the index.routing.allocation.total_shards_per_node constraint when indices are under heavy load, to prevent too many hot shards from being allocated to the same node. We are contemplating generalising this to a more nuanced set of per-index resource constraints too (#17213 (comment)).

If an index is managed by ILM then it is likely its resource requirements will change along with its lifecycle. In particular when the index leaves the hot phase it will no longer see such heavy indexing, and so any total_shards_per_node constraint is likely no longer necessary. It's important to keep the use of this sort of setting to a minimum to avoid over-constraining the allocator since this can lead to unassigned shards (#12273). I therefore think it'd be a good idea to allow this kind of setting to be changed in the appropriate ILM phase transitions.

[elasticmachine]

Pinging @elastic/es-core-features

[elasticmachine]

Pinging @elastic/es-distributed

[DaveCTurner]

I raised this question with the distrib team today and although we're not super-keen on recommending the widespread use of the total_shards_per_node constraint, we think it's a good idea to allow it to be relaxed on ILM phase transitions, particularly the hot-to-warm one.

[vigyasharma]

Are per-index resource constraints being planned as prescriptive - as in allocator makes a best effort to comply by constraints but allocates the shards anyway if no node is able to fulfill its requirements, or will it be a hard limit, and leave shards unallocated if not fulfilled?

Resource requirements are often quite dynamic. Even in the hot stage, and index may have off peak hours when the reserved resources could've been shared by other indices.

Requirement prediction itself is hard with users almost always under/over provisioning in the first pass. It needs iterative fine tuning to get the limits right. I would vote for a prescriptive best case effort to allow for such flexibility.

Separately, +1 on revising constraints through ILM transitions.

[dakrone]

We discussed this and agreed on putting an option for setting index.routing.allocation.total_shards_per_node into the allocate action in ILM.

[avneesh91]

@DaveCTurner would it be ok if I started working on this?

[dakrone]

@avneesh91 sure if you'd like to contribute a PR for this that would be great.