Cluster will not automatically assign shards

[alicegoldfuss]

Elasticsearch version: 2.2.1

JVM version: java version "1.8.0_31"
Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode)

OS version: CentOS release 6.7 (Final)

Description of the problem including expected versus actual behavior:

Thanks in advance for your help.

I inherited a poorly-behaving Elasticsearch cluster that I've been tuning on and off for a few months. Some things have improved, others have not.

About a month ago, another team needed to reboot the (AWS hosted) nodes. I turned off shard allocation in the cluster with an API call and got a success response. However, it didn't take. So I also turned off allocation in kopf. But, rebooting a node caused shards to be allocated anyway. So we rebooted one box per day and let the cluster rebalance each time.

Some time after the reboots, I noticed the cluster was still yellow. It had stopped allocating shards. I figured the API call had finally taken, so I turned allocation back on (and got another success message). According to kopf, allocation is enabled. But the cluster did not automatically assign shards.

I tried toggling shard allocation on and off with both kopf and the API. But the cluster would not allocate shards.

I complained about this on Twitter and an Elastic employee recommended upgrading. So, I upgraded the cluster from 1.5.2 to 2.2.1. However, the behavior persists.

If I assign shards manually with the API, they allocate just fine.

Steps to reproduce:

1. Turn off shard allocation with API call and kopf
2. Turn on shard allocation with API call and kopf
3. Cry a lot

Provide logs (if relevant): I didn't see any relevant logs, but if you tell me what to look for I can grep. I'm happy to gather diagnostics for you, but please note that I no longer own this cluster as of 3pm PDT Friday.

[clintongormley]

Hi @alicegoldfuss

Could you provide the actual command that you use to enable/disable shard allocation, along with the output of:

GET _cluster/settings

and (assuming you have unassigned shards atm):

GET _shard_stores

Also, would be good to take an unassigned shard and try to assign it to a particular node with the cluster-reroute API, adding the ?explain flag to figure out why the shard isn't being assigned.

thanks

[alicegoldfuss]

Hi Clinton!

$ curl -XPUT http://node-1.com:9200/_cluster/settings -d '{"transient":{"cluster.routing.allocation.enable": "all"}}'
{"acknowledged":true,"persistent":{},"transient":{"cluster":{"routing":{"allocation":{"enable":"all"}}}}}

$ curl -XGET http://node-1.com:9200/_cluster/settings
{"persistent":{"cluster":{"routing":{"allocation":{"enable":"all"}}}},"transient":{"cluster":{"routing":{"allocation":{"enable":"all"}}}}}

Shard stores returned a lot, so I cleaned up one of the replies:

{"indices":{"index-1":{"shards":{"2":{"stores":[{"urZEQshBQourcmigNqeG0g":{"name":"node-1","transport_address":"127.0.0.1:9300","attributes":{"ec2az":"us-east-1a","datacenter":"use1v"}},"version":40,"allocation":"primary"},{"uWmawPeUQ2KYNR1sLKvxfA":{"name":"node-2","transport_address":"127.0.0.1:9300","attributes":{"ec2az":"us-east-1d","datacenter":"use1v"}},"version":25,"allocation":"unused"}]}}},

The explain query dumped a bunch of stuff. Can I give it to you privately?

[alicegoldfuss]

Also I should add that assigning the shard via the API worked. That has always worked. But the cluster doesn't assign automatically.

[clintongormley]

The explain query dumped a bunch of stuff. Can I give it to you privately?

Sure. clinton at elastic dot co

Also I should add that assigning the shard via the API worked. That has always worked. But the cluster doesn't assign automatically.

Do you have ongoing recoveries? What is the output of:

GET _cat/recovery?v

I also see you're using node attributes, and possibly multiple data centres? (usually this is a no-no, unless they're like ec2 availability zones, ie as fast as a lan). Could you paste the settings you're using for shard awareness/allocation?

[alicegoldfuss]

I will send you the query dump and recovery dump.

We are using multiple AWS availability zones, not physical data centers.

[clintongormley]

@alicegoldfuss It looks like you tried to reroute an already assigned shard. Could you try one that is unassigned, eg shard 1 of index logstash-2016.02.13? (with ?explain)

[clintongormley]

Also, could you paste the settings you're using for shard awareness/allocation?

[alicegoldfuss]

@clintongormley I assigned a shard that was unassigned. But when I reran the command to create that file, yes it was already assigned.

I will send you the new shard assignment.

Can you tell me where to find the settings for shard awareness/allocation?

[clintongormley]

@alicegoldfuss Either in index settings or in node settings, depending on what is set. Try:

GET _nodes/settings
GET logstash-2016.02.13/_settings

[alicegoldfuss]

Okay, emailed those settings to you.

[s1monw]

@alicegoldfuss apparently those shards can be allocated so if you just call _reroute without any body or arguments (now that I think about it you might need to specify an empty body) will shard get allocated? if you allocate one shard manually will others follow once that shards is started?

[alicegoldfuss]

Hello!

If I manually allocate one shard, the other shards will not follow suit.

I ran the following:

curl -XPOST 'http://node-1.com:9200/_cluster/reroute'

That returned a large response, full of shard states:

{"state":"STARTED","primary":false,"node":"XXXXXXXXXXX","relocating_node":null,"shard":4,"index":"index-1","version":24,"allocation_id":{"id":"XXXXXXXXXXXX"}}

But no signs of actual allocation.

I ran it with an empty body and got the same.

[s1monw]

well I think from here on I don't have many more ideas than enabling trace logging for allocation for cluster.routing.allocation.decider as well as cluster.routing.allocation @bleskes any ideas?