Allocation filtering works only on delta settings - old excluded attributes are not honored

[malpani]

Elasticsearch version (bin/elasticsearch --version): 7.1, 7.4 and likely other versions

Description of the problem including expected versus actual behavior:
Allocation filtering works only on delta settings - ie. old filter attributes of the same family(include/exclude/require) are not honored

This bug occurs because FilterAllocationDecider is recreating its include/exclude/require filters from the settings update consumer. The AffixMapUpdateConsumer is only passing setting deltas

So with 2 independent exclusions, we end up having a cluster where /_cluster/settings has multiple attributes for shard exclusion but the decider acts only on few of them which were part of the last exclude delta

This does not look like expected behavior.
Till we have a fix, I would also like to get your thoughts on updating https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-filtering.html to indicate such cases

Steps to reproduce:

1. Decommission a zone - relocate shards from all nodes in a particular zone

curl -H "Content-Type: application/json" -XPUT localhost:9200/_cluster/settings -d '{
    "transient" : {
        "cluster.routing.allocation.exclude.zone" : "us-east-1a"
    }
}'

If this is the only excluded attribute, everything works as expected

2. Decommission a rack - relocate shards from all nodes in a particular rack

curl -H "Content-Type: application/json" -XPUT localhost:9200/_cluster/settings -d '{
    "transient" : {
        "cluster.routing.allocation.exclude.rack" : "rack-id"
    }
}'

3. At this point, the cluster settings will have both

"cluster.routing.allocation.exclude.rack" : "rack-id"
"cluster.routing.allocation.exclude.zone" : "us-east-1a"

But because the delta on second update was only for rack, shards will start relocating into the excluded us-east-1a zone

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Settings)

[DaveCTurner]

I confirmed this affects versions all the way back to 6.1.0 (6.0.0 is unaffected) so possibly #26819 is related?

I think this is a duplicate of the issue that @sohami was experiencing at #55764, see https://discuss.elastic.co/t/filterallocationdecider-routing-exclude-setting-behavior/229800 for further discussion on that subject. Thanks to both @sohami and @malpani for reporting this.

[wdongyu]

I come up with a thought that we can merge the incoming include/exclude/require filters with the existing ones, and use the new merge filters to build cluster include/exclude/require filters.

[wdongyu]

Currently allocation filtering works only on delta settings. For example:

1. we first set cluster.routing.allocation.exclude.zone to us-east-1a to allocate shards to all nodes not in zone us-east-1a.
2. Then we update cluster.routing.allocation.exclude.rack to rack-id. At this point, shards will relocate to nodes not in rack rack-id, ignoring the zone tag which we first set.

The reason for this bug is, when updating cluster settings for cluster require/include/exclude filters, AffixMapUpdateConsumer is only passing setting deltas, then FilterAllocationDecider rebuilds new filters using the settings deltas.

To fix this, we can merge the incoming include/exclude/require delta filters with the existing ones, and use the new merge filters to build cluster include/exclude/require filters. In this way, cluster require/include/exclude filters will include all cluster settings we have set instead of the delta ones.

PR: #91582