Improve handling of incompatible snapshots during restore

[ppf2]

Related to #20866 but on the functionality side.

A snapshot can contain multiple indices that are created across different major versions, eg. a snapshot containing an index created on 1.x (upgraded to 2.x), and an index created on 2.x.

When attempting to restore such snapshot to a 5.0 cluster, it currently (correctly) checks and rejects the snapshot request when encountering a snapshot that is created in 1.x:

      "type": "illegal_state_exception",
      "reason": "The index [[.marvel-2016.08.25/WL796FFwR76Zas-sILNlMQ]] was created before v2.0.0.beta1. It should be reindexed in Elasticsearch 2.x before upgrading to 5.0.0-beta1."

It will be nice to provide additional features:

• Provide a way for users to easily tell if a snapshot contains previously created indices. Currently, the snapshot _status API does not include index.version.created information. It will be nice to include index.version.created and index.version.upgraded as metadata as part of the _status API.
• Currently snapshot restore fails right away when the list of indices requested contains ones created in versions that it does not support/not compatible with (eg. if a snapshot contains both indices created on 1.x and 2.x, the restore will fail completely). It will be nice to provide an option to ignore these 1.x snapshots (eg. a flag like ignore_incompatible, ignore_obsolete, etc..).

[liketic]

@ppf2 @clintongormley Is this still worth to fix? Thanks in advance.

[clintongormley]

@tlrx do you know if this has already been fixed?

[tlrx]

@clintongormley None of the two suggested improvements have been implemented.

Provide a way for users to easily tell if a snapshot contains previously created indices. Currently, the snapshot _status API does not include index.version.created information. It will be nice to include index.version.created and index.version.upgraded as metadata as part of the _status API.

The versions are already read while building the output of the snapshot status response but they are not used and propagated to the response. I agree that they are useful information so I think we can add them under a settings object (located the same level as shards_stats, stats and shards) for each index in the snapshot:

{
  "snapshots" : [
    {
      "snapshot" : "snap-5.6.4-upgraded",
      ...
      "indices" : {
        "books" : {
          "settings" : {
                "index.version.created":"2040699",
                "index.version.upgraded":"5060499"
          },
          "shards_stats" : {
            ...
          },
          "stats" : {
            ...
          },
          "shards" : {
            ...
          }
        },

As we are here, we could also show the full index settings that contain creation_date, number_of_replicas, number_of_shards, uuid etc. It adds verbosity but it can help administrators to know if they need to override some setting.

Currently snapshot restore fails right away when the list of indices requested contains ones created in versions that it does not support/not compatible with (eg. if a snapshot contains both indices created on 1.x and 2.x, the restore will fail completely). It will be nice to provide an option to ignore these 1.x snapshots (eg. a flag like ignore_incompatible, ignore_obsolete, etc..).

I agree such an option could be useful but I'm worried by the side effects, specially when the global state is restored too. I think it's doable but I'd prefer to know what @imotov thinks about this before moving forward - and I'm not sure this one is a good adoptme candidate.

[imotov]

I think this is needed if we want to have the same repository shared between major version. Saying that, my opinion didn't change, I would like to get to the state where the repo, created with the previous major version of elasticsearch becomes read-only and we start a fresh repo after each major version upgrade. I think, that's the only way to keep things sane upgrade-wise. If such model is implemented all these flags became unnecessary.

[clintongormley]

I would like to get to the state where the repo, created with the previous major version of elasticsearch becomes read-only and we start a fresh repo after each major version upgrade

💯

[ywelsch]

I've discussed this with @tlrx and we decided not to take action on this. Given that we have the upgrade assistant today which will identify the indices that are out-of-date before an upgrade, we think that the benefits of adding the suggested functionality here are rather small, and have the down-side of convoluting the snap/restore APIs. I'm therefore closing this issue for now, but I'm open to reopen it if there's renewed interest in this.

[original-brownbear]

I'd like to reopen this, given the current work on cleaning up unreferenced data.
I think we should take some action here to:

1. Either clean up snapshots that are definitely not restorable anymore (segment files are too old). Currently these are simply dead data in the repository because they can't even be restored by using an older ES versions if another snapshot happened as the older version will fail to deserialize the most recent index-N blob.
2. Ensure that at least an older Elasticsearch version will still be able to restore the unreferenced indices.

I think the second option is a lot safer than the first so I'd prefer it. I think we could implement this relatively easily going forward at least (this wouldn't help those that already ran into this and have dead 2.x data in a 6.x+ cluster) by:

Simply using a new naming schema for index-N (BlobData) blobs at the repository root for every new major version. Could use something like index7-N, index8-N and so on at the repo root as well as the shard level index-N blobs. Then whenever we upgrade to a newer ES version we simply leave the latest version for each V of those index{V}-{N} behind, allowing older versions of ES to still restore these snapshots if needed (we would probably also want to add code to prevent older versions from writing to the repo if a newer index{V} blob is present).

Long story short: let's either delete unusable data blobs are keep them useful, but not keep collecting stale blobs on every update if snapshots aren't deleted.

WDYT?