Write index metadata on data nodes where shards allocated

[bobpoekert]

If the master node thinks that an index does not exist, and another node thinks that it does, the conflict is currently resolved by having the node that has the index delete it. This can easily result in sudden unexpected data loss. The correct behavior would be for the conflict to be resolved by both nodes accepting the state of the node that thinks that the index exists.

[vjanelle]

Are you running dedicated masters with no query/data load?

[bleskes]

@bobpoekert can you elaborate more about what happened before the data was lost - did you update the mapping (title suggest so)? If you can share your cluster layout (dedicated master nodes or not) it would be great. Also please the grab a copy of the logs of the nodes and save them. They might give more insight.

[bobpoekert]

@vjanelle yes

@bleskes The sequence of events was the following:

1. Remove node (which is a master candidate) from cluster
2. Delete all index files from said node
3. Add node back into cluster
4. Node is elected master
5. All indexes in the cluster are now gone

[vjanelle]

Did you have data turned off on the master candidate as well in the configuration?

[bobpoekert]

@vjanelle No. The master candidate is also a data node.

[bleskes]

@bobpoekert thx. let me make sure I understand what you are saying:

1. when you started you had a master node running, but it was also allowed to have data (i.e., it didn't have node.data set to false in its elasticsearch.yml file). Call this node A.
2. you brought down another node which was both a master candidate and a data node (i.e., neither node.master nor node.data was set to false). Call this node B.
   3).While B was down, you delete it's data folder (right? or was it some sub folders of it).
3. You brought B back up and it joined the cluster.

From this point on I'm not clear. Why was node B elected as master? What happened to the residing master node A?

[bobpoekert]

@bleskes

1. Cluster has a single master node (A) and two data nodes (B and C)
2. Remove A from the cluster
3. Now the cluster is offline (has no master)
4. Delete A's data folder
5. Bring A back up
6. Now B and C have no data

[bleskes]

@bobpoekert I'm confused. You said before you had nodes that are master candidates and are also data nodes. Can you confirm that node A has node.data: false in it's settings and and that node B & C have node.master: false ?

[bobpoekert]

@bleskes
All the nodes have data: true.
A has master: true
B and C have master: false

[bleskes]

OK. Clear. The cluster meta data (which we call cluster state) is stored and maintained on the master nodes. That meta data contains which indices are out there in the cluster. We only write the metadata on master eligible nodes, and rely on multiple masters for redundancy of this data set (compared to specific shard data).

Since you only have one master node, that means there is no redundancy in your cluster meta data storage. After you deleted it there is nowhere to get it back from so the cluster becomes empty.

We do have a feature that is called dangling indices, which is in charge of scanning data folders for indices that are found on disk but are not part the cluster state and automatically import then into the cluster. As it is today, this feature needs to find some part of the index meta data to work, but those are also stored on the master eligible nodes, which in your case there were none.

Thinking about it, we can be more resilient in situations where users are running only a single master node (though we highly recommend running more than one), and store the index metadata wherever a shard copy is stored, so also on data nodes. So we can improve the dangling indices case to identify those as well.

Lets keep this issue open and we will work on a PR to improve things based on the above.

[lusis]

This definitely feels like a documentation issue as well. I asked on twitter and the reason for running single master was essentially to avoid other bugs. I don't think it's an unfair assumption that a user would expect having a quorum of preexisting data nodes to be enough to promote a new master or rebuild it without data loss. I would, from a purely semantic perspective, expect that data nodes would have ALL the data needed for the cluster and that the master's state would live with the rest of the "data"

Also would it not make sense for maybe non-master eligible nodes to at least provide a backup of the master node cluster metadata for this case and as a safety precaution?

Ftr, I have no direct impact from this issue. Just another production ES user who tracks this stuff.

[lusis]

Sorry missed the part where you mention possibly storing the backup on data nodes.

[grantr]

Repro of this bug: https://gist.github.com/grantr/a53a9b6b91005ad9807f

This is more than a documentation issue. Even when running in a degraded configuration, shards should never be deleted if their metadata can't be found.

[bleskes]

Also would it not make sense for maybe non-master eligible nodes to at least provide a backup of the master node cluster metadata for this case and as a safety precaution?

This is indeed the plan