Restoring ILM managed indices uses original indices' age for deletion

[dakrone]

Consider the following (simplified) use case, you want to retain indices for a
period of time, and during that time, you take periodic snapshots of the
indices:

(30 seconds is a stand in for a longer time, to make this more easily
reproducible, pretend it is 90d or something for a production use case)

PUT _ilm/policy/delete-old
{
  "policy": {
    "phases": {
      "delete": {
        "min_age": "30s",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

And create an index using that policy

PUT /foo
{
  "settings": {
    "index.number_of_shards": 2,
    "index.lifecycle.name": "delete-old"
  }
}

After the index is created, during the periodic snapshotting, a snapshot is
taken:

PUT /_snapshot/backup
{
  "type": "fs",
  "settings": {
    "location": "/tmp/foo"
  }
}

PUT /_snapshot/backup/snap1?wait_for_completion=true

Next, after 30 seconds has elapsed, the index foo is deleted.

Later, a user decides they need the data on the "foo" index and thus need to
restore the index:

POST /_snapshot/backup/snap1/_restore

The index is restored... and then immediately deleted because it is older than
30 seconds.

[elasticmachine]

Pinging @elastic/es-core-infra

[colings86]

It’s not obvious that the above isn’t what the user wants.

In a scenario where the user is doing disaster recovery from a snapshot the user will want the index to maintain the state from ILM so that after disaster recovery the index continues as if it were the original copy.

But in a situation where the user is wanting to restore an old index to read the data again the user may want the index to persist so they can access the data even though the index is past the deletion age.

So this isn’t as straight forward as changing the creation timestamp on restore.

[colings86]

This may actually also be an issue in CCR since when restoring an index that was a following index you may not want it to try to follow the leader index anymore. I wonder if this is a scenario that has been solved in CCR and we can use the same approach or whether there is a common approach we should adopt for these kinds of features?

/cc @jasontedor @martijnvg

[dakrone]

We discussed this and for now a workaround is setting index.lifecycle.name to null when doing the restore. We also discussed bringing back the index.lifecycle.skip setting to allow skipping a particular index, which could be set to true during the restore.

Regardless, we should document this as a known issue / workaround for now with intent to find a more elegant solution after the initial release.

[dakrone]

This has been resolved by a documentation change. However, we'd like to address this in a better way in the future, so we're leaving this open for now.