[ILM] rollover + `settings.index.priority`

[jpcarey]

Describe the feature: It seems like setting the index.priority on the underlying indices so that the current index (rollover, daily?) would always be prioritized for recovery.

It could be set via the template, but then it needs to be lowered as new indices are created.

https://www.elastic.co/guide/en/elasticsearch/reference/6.5/recovery-prioritization.html

[elasticmachine]

Pinging @elastic/es-core-features

[jakelandis]

@jpcarey - wouldn't the implicit priority for roll over always be the current index ?

From https://elastic.co/guide/en/elasticsearch/reference/6.5/recovery-prioritization.html :

This means that, by default, newer indices will be recovered before older indices.

Can you describe your use case a bit more (or help me understand why the implicit recovery order isn't sufficient).

[jpcarey]

@jakelandis in the case of multiple rollover indices, you can have different size/document/age requirements for each rollover setup. This means you could have newer rolled over indices from rollover setup "A", vs the current target index setup for rollover "B".

[jakelandis]

Chatted with @jpcarey on this for a bit and the crux of the issues is to help ensure that current index, across all indexes, are prioritized for recovery.

For example, if you have a logs index that rolls every hour, and monitoring index that rolls every day, then you can have ~23 indexes (logs) that are newer than the current writable monitoring index. When recovery happens, those 23 old indexes will get prioritized before the 1 writable monitoring index.

The change here could be to add a high priority to current writable index and remove it when it rolls over or decrease the priority when it rolls over (assuming the the priority is already set). The goal to ensure/enable the ability to allow the current writable indexes that are managed by ILM to recover first.

Proposal:
Add an index_priority option to hot, warm, and cold phases such that when an index enters that phase that priority is assigned to the index.

Thoughts?

[jakelandis]

We discussed this abit and will implement this as a new action available for the hot, warm, and cold phases.

The tentative API will look like this: (where the set_priority will always execute as the first action if multiple actions are defined)

PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_age": "10m"
          }
        }
      },
      "warm": {
        "min_age": "1m",
        "actions": {
          "set_priority": {
            "priority": 90
          }
        }
      }
    }
  }
}