ILM: IndexLifecycleRunner should not spam logs with warnings when policy is misconfigured

[talevy]

When an index sets its index.lifecycle.name to an invalid one

• The system will make it look like things are being managed correctly... when really there is no real policy of that name
• the IndexLifecycleRunner will spam the logs looking for the first step with messages of this nature:

[WARN ][o.e.x.i.IndexLifecycleRunner] [node-0] current step [null] for index [thingy-000002] with policy [thingy] is not recognized

reproduce with

PUT _template/thingy
{
   "index_patterns": ["thingy*"],
   "settings": {
     "index": {
       "lifecycle": {
         "name": "thingy",
         "rollover_alias": "thingy-alias"
       },
       "number_of_shards": "1",
       "number_of_replicas": "1"
     }
   }
}

PUT /_ilm/policy1
{
  "policy": {
    "phases": {
      "hot": {
        "after": "0s",
        "actions": {
          "rollover": {
            "max_docs": 1
          }
        }
      }
    }
  }
}

PUT /thingy-000001
{
  "aliases": {
    "thingy-alias": {
      "is_write_index": true
    }
  }
}

POST thingy-000001/_doc
{
  "foo": "bar"
}

GET thingy*

It is a very simple thing to happen, and the logs are not useful in helping figure out what is going on

To add to the confusion, here is the output of some APIs we have to help explain what is happening to users when things aren't working properly

GET thingy-000002/_ilm/explain

{
  "indices": {
    "thingy-000002": {
      "index": "thingy-000002",
      "managed": true,
      "policy": "thingy",
      "skip": false,
      "lifecycle_date": -1,
      "phase": "",
      "phase_time": -1,
      "action": "",
      "action_time": -1,
      "step": "",
      "step_time": -1
    }
  }
}

We should provide better feedback when invalid policies are set on indices.

[elasticmachine]

Pinging @elastic/es-core-infra

[dakrone]

We discussed this and decided to keep the index as managed, and move it to an error state with the message that we don't have a policy to manage the index with. The log message should be moved to TRACE rather than WARN

[gwbrown]

Resolved by #34206