Updating ILM policy for an index while in the same phase does not respect new policy's same phase's min_age

[dakrone]

If I have two competing policies:

PUT _ilm/policy/my_lifecycle3
{
  "policy": {
    "phases": {
      "warm": {
        "min_age": "30s",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          },
          "shrink": {
            "number_of_shards": 1
          },
          "allocate": {
            "include": {
              "type": ""
            },
            "exclude": {},
            "require": {}
          }
        }
      },
      "cold": {
        "min_age": "60s",
        "actions": {
          "allocate": {
            "number_of_replicas": 2,
            "include": {
              "type": "cold"
            },
            "exclude": {},
            "require": {}
          }
        }
      },
      "delete": {
        "min_age": "2m",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

And an alternate policy:

PUT _ilm/policy/newpolicy
{
  "policy": {
    "phases": {
      "warm": {
        "min_age": "100m",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          },
          "allocate": {
            "number_of_replicas": 2,
            "include": {
              "type": "cold"
            },
            "exclude": {},
            "require": {}
          }
        }
      },
      "delete": {
        "min_age": "200m",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

And I create an index with the first policy (where the "warm phase kicks in at
30 seconds"):

PUT /foo
{
  "settings": {
    "index.lifecycle.name": "my_lifecycle3",
    "index.number_of_shards": 2
  }
}

Then wait at least 30 seconds until the "foo" index is in the
warm/complete/complete action (meaning everything in the warm phase has
finished, the index is between 30 and 60 seconds old)

Next, I update the policy for the "foo" index to use the "newpolicy", which has
a "warm" phase with a min_age of 100 minutes:

PUT /foo/_settings
{
  "index.lifecycle.name": "newpolicy"
}

The step info for the foo index is still at warm/complete/complete, meaning
that it has now skipped the steps that would normally occur in the newpolicy
"warm" phase, even though the index is not old enough yet to transition into the
"warm" phase with the new policy.

[elasticmachine]

Pinging @elastic/es-core-infra

[colings86]

I don't see this as a bug. This is in line with the explanation of how policies work. A policy will enter a phase when the min age is met (the original one) and will progress through the phase, if the policy changes we do not pick up on the changes until the next phase so I do not expect the phase to be re-run if the min_age changes. This would be extremely difficult to implement in any case because the shrink action would not be able to work if it is run multiple times and even if it could it would open weird, hard to resolve edge cases where e.g. the index start with 6 shards and in the original policy is shrunk to 1 shard but in the updated policy is shrunk to 2 shards, in this case the index would get stuck because of re-running the warm phase.

I think its quite reasonable and easy to explain that the policy is run forward-only and once we have started doing something (like a phase) we won't go back and run it later.

[dakrone]

I think its quite reasonable and easy to explain that the policy is run forward-only and once we have started doing something (like a phase) we won't go back and run it later.

The issue with this is that the behavior between changing an index's policy (from my_lifecycle3 to newpolicy) is different than removing my_lifecycle3 and then setting the policy to newpolicy. To me that seems like a bug, since I would expect changing the policy to be the same as removing the original and then setting the new policy.

[colings86]

The remove API is not removing the policy though, its clearing IM entirely form the index, so its never going to be the same as setting a different policy. Setting the index to a different policy is equivalent to clearing the index.lifecycle.name setting and then re-setting it.

[dakrone]

We discussed this and decided that there's not really a good way to satisfy the issue where we don't want to re-run phases (for example, we can't re-rollover or re-shrink once we've done those things). As that is the case, we're going to document the existing behavior (probably alongside the documentation about phase JSON caching).

I'll change this from a bug to a documentation task

[jakelandis]

Should add some documentation to https://www.elastic.co/guide/en/elasticsearch/reference/6.x/_updates_to_executing_policies.html