Searchable snapshot index remains green when backing snapshot is deleted

[n0othing]

Elasticsearch Version

8.1.2

Installed Plugins

No response

Java Version

bundled

OS Version

ESS

Problem Description

Today it's possible to delete the backing snapshot of a searchable snapshot index without deleting the index itself [1]. When this happens, the index reports as green/healthy until the shard becomes de-allocated (e.g restarting the hosting node). The safeguard #73821 will be very helpful, but the index shouldn't report as healthy since it's not fully searchable (I believe you can still retrieve anything cached, but that's it).

[1] #73821

Steps to Reproduce

# Set ILM poll interval to 10s

PUT _cluster/settings
{
  "persistent": {
    "indices.lifecycle.poll_interval": "10s"
  }
}


# Create ILM policy

PUT _ilm/policy/test-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_docs": 3
          },
          "set_priority": {
            "priority": 100
          }
        },
        "min_age": "0ms"
      },
      "frozen": {
        "min_age": "30s",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "found-snapshots"
          }
        }
      }
    }
  }
}

# Create index policy w/ ILM policy attached.

PUT _index_template/test-template
{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "test-policy",
          "rollover_alias": "test"
        },
        "number_of_replicas": "0"
      }
    }
  },
  "index_patterns": [
    "test-*"
  ],
  "composed_of": []
}

# Bootstrap the first index

PUT test-000001
{
  "aliases": {
    "test": {
      "is_write_index": true
    }
  }
}

# Index some data

POST _bulk?refresh=wait_for
{ "index" : { "_index" : "test" } }
{ "field" : "Hello World!" }
{ "index" : { "_index" : "test" } }
{ "field" : "Hello World!" }
{ "index" : { "_index" : "test" } }
{ "field" : "Hello World!" }

# Confirm rollover

GET _cat/indices/test*?v

# Delete the backing snapshot for partial-test-000001

DELETE _snapshot/found-snapshots/2022.04.05-test-000001-test-policy-zwqycl2cti6qdgimosrceg

# Index still shows as green

GET _cat/indices/test*?v

health status index               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   test-000002         bpYGqHYqR4qVFmG2IolSjg   1   0          0            0       225b           225b
green  open   partial-test-000001 CyKSzRexRs-oDJD8ZU2E8g   1   0          3            0         0b             0b

Logs (if relevant)

No response

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)