Incorrect documentation about multiple awareness attributes

[bittusarkar]

Elasticsearch version: 5.4.1

Plugins installed: N/A

JVM version: 1.8.0_72

OS version: Windows 10

Description of the problem including expected versus actual behavior: The documentation here is misleading. Copying it below:

Multiple awareness attributes can be specified, in which case the combination of values from each attribute is considered to be a separate value.

Steps to reproduce:

1. Set up an Elasticsearch cluster with three nodes, say, n1, n2 and n3 with the following configuration:

• elasticsearch.yml of n1:

cluster.name: c2
node.name: n1
cluster.routing.allocation.awareness.attributes: updateDomain, faultDomain
cluster.routing.allocation.awareness.force.updateDomain.values: 0,1
cluster.routing.allocation.awareness.force.faultDomain.values: 0,1,2
node.attr.updateDomain: 1
node.attr.faultDomain: 0

• elasticsearch.yml of n2:

cluster.name: c2
node.name: n2
cluster.routing.allocation.awareness.attributes: updateDomain, faultDomain
cluster.routing.allocation.awareness.force.updateDomain.values: 0,1
cluster.routing.allocation.awareness.force.faultDomain.values: 0,1,2
node.attr.updateDomain: 0
node.attr.faultDomain: 1

• elasticsearch.yml of n3:

cluster.name: c2
node.name: n3
cluster.routing.allocation.awareness.attributes: updateDomain, faultDomain
cluster.routing.allocation.awareness.force.updateDomain.values: 0,1
cluster.routing.allocation.awareness.force.faultDomain.values: 0,1,2
node.attr.updateDomain: 1
node.attr.faultDomain: 2

2. Create a test index:

PUT idx1
{
  "settings": {
    "number_of_shards": 1, 
    "number_of_replicas": 1
  }
}

3. Shards should be allocated on either n1 and n2 or n2 and n3. Say, the primary is allocated on n1 and replica is allocated on n2. Try moving the replica to n3:

POST _cluster/reroute?explain
{
  "commands": [
    {
      "move": {
        "index": "idx1",
        "shard": 0,
        "from_node": "n2",
        "to_node": "n3"
      }
    }
  ]
}

It will fail with the below explanation:

{
  "decider": "awareness",
  "decision": "NO",
  "explanation": "there are too many copies of the shard allocated to nodes with attribute [updateDomain], there are [2] total configured shard copies for this shard id and [2] total attribute values, expected the allocated shard count per attribute [2] to be less than or equal to the upper bound of the required number of shards per attribute [1]"
}

What this means is that for multiple awareness attributes, each on its own is considered to be a unique value and not a combination of them. If combination was considered to be unique, then the cluster reroute should have worked but it did not.

[elasticmachine]

Pinging @elastic/es-distributed

[DaveCTurner]

Thanks for the detailed instructions. A slightly simpler test is not to involve node n2 at all, which is almost in the test suite:

--- a/server/src/test/java/org/elasticsearch/cluster/routing/allocation/AwarenessAllocationTests.java
+++ b/server/src/test/java/org/elasticsearch/cluster/routing/allocation/AwarenessAllocationTests.java
@@ -834,7 +834,7 @@ public class AwarenessAllocationTests extends ESAllocationTestCase {
         nodeAAttributes.put("rack", "c");
         Map<String, String> nodeBAttributes = new HashMap<>();
         nodeBAttributes.put("zone", "b");
-        nodeBAttributes.put("rack", "d");
+        nodeBAttributes.put("rack", "c");
         clusterState = ClusterState.builder(clusterState).nodes(DiscoveryNodes.builder()
             .add(newNode("A-0", nodeAAttributes))
             .add(newNode("B-0", nodeBAttributes))

This seems like the desired behaviour so I agree it's the docs that need fixing.