Tried to recover failed shards of index endlessly

[mimiyan]

I saw other users reported the same issue at #2598 and the developers suggested to create a new issue for this., but I did not find a new ticket is created. So I created this one.

When a shard of index fails during recovery, ElasticSearch keeps to recover the same shard every 10 seconds. It would be nice for users to configure the number of times to try to recover instead of trying it forever.

[kimchy]

@mimiyan which version do you witness this behavior with? It should be much better in 1.6.

[clintongormley]

@mimiyan also what errors are you seeing in the logs?

[mimiyan]

I am using version 1.5.

The logs show:
2015-05-22 08:32:25,311][WARN ][indices.cluster ] .....marking and sending shard failed due to [failed recovery]
[2015-05-22 08:32:25,312][WARN ][cluster.action.shard ] [node1] [messages-20150509][0] received shard failed for [messages-20150509][0], node[8YxdI8jsThuwjUk1qzjvaA], [P], s[INITIALIZING], indexUUID [bHbn-05QSYSB9GTuZxZjHA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[messages-20150509][0] failed recovery]; nested: EngineCreationFailureException[messages-20150509][0] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/data/.../index/segments_6")]; ]]

The above messages keep popping up every 10 seconds.

[bleskes]

@mimiyan how many nodes do you have in the cluster? If there is no other node to put the shard, I suspect you run into #10558 ...

[mimiyan]

@bleskes We only have 1 node in our cluster now.

[bleskes]

@mimiyan I see. In this case the no other options and there is no way to recovery. I agree we shouldn't spin with endless attempts. A temporary work around would be to move the corrupted shard folder. Note that your index will stay red as they is no other copy to use. You can issue a manual _reroute command to force allocation of a new empty primary.

[bleskes]

at second thought - #11269 (released in 1.6.0) should fix this (we will no longer think the shard copy is good during assignment, only to discover it's bad in recovery). I'm going to close this as we got to the root cause of the matter. If upgrading to 1.6.0 somehow doesn't help please do reopen.