Data loss when old master node dead and startup again

[jffree]

Describe the feature:

Elasticsearch version (bin/elasticsearch --version): elasticsearch/elasticsearch-oss:6.3.2

docker version: 18.03.1-ce

OS version (uname -a if on a Unix-like system): centos7.5

Description of the problem including expected versus actual behavior:

Steps to reproduce:

1. we have two node cluster, A (master), B(slave)
2. A node dead, and B become new master (standalone)
3. Post new data to B
4. A startup, and join cluster, B is still as a master, But new data lost, all data become old.

[tvernum]

@jffree The idea of a "slave" doesn't exist in Elasticsearch. We have "master eligible" and "non master eligible" and we have an "elected master", but "slave" isn't a term we use so it's not clear exactly what your configuration was.

It sounds like you might have been running with an unsafe configuration - can you please update your desciption to use clear Elasticsearch terminology so we can be sure what your setup looked like.

We need to know:

• how many nodes you had
• which of those nodes were master eligible. That is, in their elasticsearch.yml you had node.master: true OR there was no setting for node.master (the default value is true)
• which node was the elected master
• when you say "B become new master" was that automatic, or did you have to make a configuration change?

[jffree]

@tvernum

1. I have two nodes in cluster (A and B)
2. old cluster A is master, I set node.master: true in elasticsearch.yml, B is set node.master: false in
   elasticsearch.yml
3. Stop A, update B config( remove node.master: false in elasticsearch.yml), so B can run standalone
4. POST new data to B node
5. Startup A node and (set node.master: false in elasticsearch.yml), B is set node.master: true in
   elasticsearch.yml, so B is master in new cluster
6. But the new data loss!

[elasticmachine]

Pinging @elastic/es-distributed

[jffree]

@tvernum
We also have tried to delete data (UUID dir on A node) when A join cluster again, but data still lost

[ywelsch]

@jffree you haven't properly configured minimum_master_nodes. Please read https://www.elastic.co/guide/en/elasticsearch/reference/current/discovery-settings.html#minimum_master_nodes and adapt your configuration. Also note that questions like this should best be first asked on https://discuss.elastic.co instead. Thank you

[jffree]

@ywelsch
I have set minimum_master_nodes 1 in elasticsearch.yml.
Also, I have node this question on https://discuss.elastic.co, but nobody can give me an answer.
And I want to ask you, if you do not clear what happened, why you close this issue?

[ywelsch]

you have two master-eligible nodes, so you need to set minimum_master_nodes to 2, i.e. (master_eligible_nodes / 2) + 1 == (2/2) + 1 == 1 + 1 == 2 according to the linked docs. More info is also here: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#split-brain

[jffree]

@ywelsch
If I have set minimum_master_nodes 2, one node dead, and the cluster will not work.
I hope the other node will still support service.

[ywelsch]

If I have set minimum_master_nodes 2, one node dead, and the cluster will not work.

yes, you cannot have high availability in a 2 node cluster. Note that with minimum_master_nodes 2, the remaining node will be available for searches, but not for writes or metadata operations (e.g. backups).

[vigyasharma]

@ywelsch While setting minimum_master_nodes to 2 will avoid this, curious to know if this expected behavior or a replication edge case?

This does not look like a split brain, at a time there was only one master. First A is master, then it is stopped and node B is made master (single node cluster), then A is made master ineligible and added back to cluster. But it seems that node A's shards override node B's shards?

Could this be an issue around allotting primary terms for shards when master B promoted its replica to primary?

How are the cluster state details from node A and node B reconciled when A joins back?
(Apologies if this should've been a forum question, asking here to use context from this issue)

[DaveCTurner]

For reference, it looks like the same question was asked on the discussion forum --> https://discuss.elastic.co/t/data-loss-when-old-master-node-dead-and-startup-again/169506/6

@vigyasharma yes please ask your question on the forum. It's a good question, but this is not the right place to answer it.