Enrich indices can clash with user allocation settings

[markharwood]

A user reported that enrich indices were not being allocated because of the following incompatible objectives:

1. The system requirement to have an enrich shard replica on every node
2. The user choice to have >1 node per host
3. The user choice to not allocate same replica shards to the same host via cluster.routing.allocation.same_shard.host

While the enrich indices were not fulfilling their objective to have a replica on every node the system is functional and there is good locality of data (each host has a copy) but the cluster state is permanently yellow. This is an annoyance from a monitoring perspective.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Allocation)

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Ingest)

[jasontedor]

There is a general problem here, not specific to the enrich indices, relating to auto-expand replicas and how it interplays with other allocation settings (e.g., #2869). Therefore, we are re-adding the distributed label, and treating this as a distributed problem.

[DaveCTurner]

We discussed this today and think it would be a good idea to ignore the same-host allocation decider and the allocation awareness decider if using auto-expand replicas with an upper bound of all. Both of these deciders aim to achieve resilience by avoiding a concentration of shards on a single failure domain (host or zone) but you can't really get more resilient than the case where there's a copy of every shard on every node, which is what 0-all does, so these deciders do not have any real benefits.

A question to the Enrich folks: does this do what you want? We were working on the assumption that you really want a copy of every shard on each node. An alternative, if cluster.routing.allocation.same_shard.host were set, would be to require a copy of every shard on every host and to route Enrich's searches to other nodes on the same host. That sounds harder, and although it avoids a proper network hop it still entails some more de/serialization than you might like.

This also may relate to system indices so I'm adding the :Core/Infra/Core label. System indices folks: will there be system indices that should have a copy on every node and, if so, does this proposal work for you?

All other feedback also welcomed.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Core)

[gwbrown]

Enrich indices will be system indices, so if Enrich requires a shard copy on every node, by definition that will be a requirement for some system indices. I believe there are some others that would benefit from that treatment as well.

How allocation is configured for system indices is something we’ve been thinking about but have not yet reached a conclusion on, in part because of radically varying requirements for different system indices. Now that we’ve gotten hidden indices fully ready to ship, we’re looking to solidify things very soon. In any event, I don’t foresee either of the solutions proposed here colliding with our plans (per our current thinking at least).

[martijnvg]

We discussed this today and think it would be a good idea to ignore the same-host allocation decider and the allocation awareness decider if using auto-expand replicas with an upper bound of all. Both of these deciders aim to achieve resilience by avoiding a concentration of shards on a single failure domain (host or zone) but you can't really get more resilient than the case where there's a copy of every shard on every node, which is what 0-all does, so these deciders do not have any real benefits.

Thanks. I was leaning in the same direction too. I think the allocation deciders that you mentioned should implement the shouldAutoExpandToNode(...) method? (similar to what a happened to filter allocation decider in #48974)

A question to the Enrich folks: does this do what you want?

Yes. Enrich was built with the assumption that not all enrichments/lookups are going to be local. Especially in the case with clusters that only have dedicated ingest nodes. We built a buffering mechanism to bundle as search requests as much as possible under a certain load.

[DaveCTurner]

Thanks. I was leaning in the same direction too. I think the allocation deciders that you mentioned should implement the shouldAutoExpandToNode(...) method? (similar to what a happened to filter allocation decider in #48974)

No, it's kind of the opposite of that. The filter allocation decider overrules auto-expand replicas, whereas I'm proposing having auto-expand replicas overrule allocation awareness/same host.