Prevent index closing while snapshot is restoring the indices

[ppf2]

On 1.7.1. This may be related to #15432 , but it is unclear in #15432 what the cause was. So I am filing a separate ticket on this.

In short, it looks like when snapshot restore is running, if the indices are closed while it is operating on a shard, it will leave the snapshot restore request in a STARTED state.

    "restore" : {
      "snapshots" : [ {
        "snapshot" : "snapshot_name",
        "repository" : "repository_name",
        "state" : "STARTED",

And shards that are in INIT state as reported by the restore request:

{
          "index" : "pricing_2015121502",
          "shard" : 1,
          "state" : "INIT"
}

So that when the end user tries to kick off another restore, it will fail:

failed to restore snapshot
org.elasticsearch.snapshots.ConcurrentSnapshotExecutionException: [repository_name:snapshot_name] Restore process is already running in this cluster
    at org.elasticsearch.snapshots.RestoreService$1.execute(RestoreService.java:174)
    at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:374)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:196)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:162)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Because only 1 snapshot restore request can be run at any point in time.

It will be a good idea for us to implement a check to prevent users from closing indices while the restore operation is working on those shards which will help prevent this type of issue.

[clintongormley]

@ywelsch please could you take a look at this

[ywelsch]

I verified the issue on master. It is present on all ES versions.

To fix this, there are two possible ways:

• Fail closing an index if there are shards of that index still being restored.
• Always let the index close operation succeed but fail the restore process of the shards of that index. This is more in line with what we currently do for index deletes that happen during the snapshot restore process.

@clintongormley wdyt?

[ppf2]

Fail closing an index if there are shards of that index still being restored.

At least for this particular use case, this option is desirable because they didn't realize that the restore is still in progress (would like the restore to proceed), and not because they want to cancel a specific restore.

[ywelsch]

@ppf2 what's your take on closing the index during snapshot (in contrast to restore)? Currently, the close operation succeeds but we fail the shards that are closed.

[ppf2]

Currently, the close operation succeeds but we fail the shards that are closed.

For this specific report from the field, the indices did close, but the restore status from the cluster state shows a shard apparently stuck in INIT state - reopening the index did not resolve the issue, but deleting the index helped get the restore out of that stuck state. So somehow the close operation did not successfully fail the shards, but left them the restore procedure in a started state, thinking that it is still initializing the restore on one of the shards.

{
          "index" : "pricing_2015121502",
          "shard" : 1,
          "state" : "INIT"
}

[ywelsch]

@ppf2 I think you misunderstood me, I'm not claiming that the close operation successfully failed the restore process. My question was related to the comment of yours saying that it would be preferable to fail closing an index if there are shards of that index still being restored. I wanted to know whether you think the same applies to the snapshot case. So my question was: Should closing an index fail while the index is being snapshotted?

My goal here is to find out what the expected behavior should be in the following situations:

• Deleting an index during snapshot operation of that index
• Closing an index during snapshot operation of that index
• Deleting an index during restore operation of that index
• Closing an index during restore operation of that index

@clintongormley care to chime in?

[ppf2]

@ywelsch Ah sorry misinterpreted :)

I am thinking for deletions, we can cancel that snapshot operation (only on shards of that index) and perform necessary cleanup so that it doesn't end up with partial files in the repository. Same thing with restore, cancel the restore of the shards for that index, and do some cleanup in the target cluster. For closing, maybe we can just prevent them from doing so and allow the snapshot or restore operation to complete? But yah, let's get input from @clintongormley @imotov

[ywelsch]

Here are my initial thoughts:

For restore:

• Closing an index that is being restored from a snapshot should fail the close operation. There is no good reason to do allow this, as closing an index that is being restored makes the index unusable (it cannot be recovered). What the user probably intends to do is to delete the index.
• Deleting an index that is being restored should cancel the restore operation for that index and delete the index. Deleting an index that is being restored would be a simple approach to cancel the restore process for that index. An open point for me is whether the restore information returned after finishing the restore process should take the deleted index into account in its statistics and status.

For snapshot:

• Closing an index that is being snapshotted should only succeed if snapshot is marked as partial. If so, index should be closed and snapshot should abort snapshotting the shards of that index. If shards of the deleted index have been already successfully snapshotted in the meantime, they will remain in the snapshot.
• Deleting an index that is being snapshotted should only succeed if snapshot is marked as partial. If so, index should be deleted and snapshot should abort snapshotting the shards of that index. If shards of the deleted index have been already successfully snapshotted in the meantime, they will remain in the snapshot.

Note: If the user does not want to wait for non-partial snapshot to finish before executing close/delete, he has to cancel the snapshot operation by deleting the snapshot in progress.