Index aliases and delete operation

[Paikan]

Suppose we use the "usersdata flow" exposed in kimchy's "Big Data, Search and Analytics" presentation to index users documents in a big oversharded index with routing+filtering aliases. Some users content are moved to their own index when becoming too big.

From the client perspective we don't know if we are manipulating a virtual index via an alias or a real index dedicated to the user and the code should be the same. So if an user is deleted and we want to remove its content the client code do something like curl -XDELETE localhost:9200/$INDEX_NAME. The power of aliasing being that the client is agnostic of knowing if it deals with a real index or an aliased one.

But with the current behavior if $INDEX_NAME is an alias the whole big oversharded index will be deleted and all users document are lost!

IMHO it is really dangerous. I was expecting a simple delete index if the $INDEX_NAME is a real index under the hood and maybe a delete by query (even if it is expensive) or at least a failure or something like that maybe configurable via a parameter if $INDEX_NAME is an alias linking to the big oversharded index.

What do you think?

[Paikan]

Hey any update on this one? @imotov are you unhappy with your proposed fix?

[s1monw]

I thought about this a bit and I tend to agree with @Paikan that this is pretty dangerous. Maybe we can add an option that allows you to specify an "on_delete" operation on the alias that is executed on a delete and if it's not present we just fail to delete? Any thoughts?

[kimchy]

We can reject an alias with a registered filter when trying to delete it. Another potential solution is that if the delete is done on an alias with a filter, we delete by query the user content, and not delete the index (that would make the implementation much more complex though...).

@imotov regarding the change, I think it makes sense to have a method on the metadata that resolves jsut aliases out of the list of indices, and do checks on it where applicable. Or somethign like the lines. Otherwise, the concreteAndXXX might get out of hand in terms of options.

[jprante]

I think the solution of @imotov to reject aliases with an ElasticsearchIllegalArgumentException is not correct. Instead, a name of an index alias should be silently skipped to maintain the idempotency of the REST DELETE operations. I suggest the following semantics:

curl -XDELETE 'http://localhost:9200/{name}' = deletes {name} if and only if {name} is a concrete index, otherwise it does nothing and it never returns an error

curl -XDELETE 'http://localhost:9200/_alias/{name}' = deletes {name} if and only if {name} is an index alias, otherwise it does nothing and it never returns an error

[kimchy]

@jprante I might be missing something, but if it is rejected, what makes it not idempotent?

[Paikan]

I think the best approach would be a delete by query because it is what the client expect at the end

[Paikan]

The @s1monw approach is also quite flexible by default we reject the delete operation on alias with a registered filter but when creating the alias we could set a boolean in the alias to specify we want to delete data if deleting an aliased index.

[jprante]

@kimchy the question is if names are mixed in one REST command, should curl -XDELETE 'http://localhost:9200/concreteindex,aliasindexname' succeed or fail? I think it shall succeed but 'aliasindexname' is ignored.

[kimchy]

@jprante understood, and thats one option, but rejecting it will not break the REST delete contract. I don't think they should succeed because you loose information on what you intentionally asked ES to do.

[Paikan]

Till this issue has not been addressed properly why not rejecting a delete on a filtered alias to avoid data loss. Maybe a message explaining that an explicit delete by query order should be use in that case could be returned?

[Paikan]

As discussed with @s1monw ping this one as I think it is still a critical issue that can lead to data loss.