Decouple ILM action execution from the user setting the policy

[inqueue]

By design, index lifecycle management performs operations as the user who last updated the ILM policy. This behavior is documented. See the original PR.

This design is causing problems when the last user to apply ILM policies loses authorization to perform index operations performed by ILM.

    "index-000061" : {
      "index" : "index-000061",
      "managed" : true,
      "policy" : "indexpolicy",
      "lifecycle_date" : "2021-04-28T21:20:27.810Z",
      "lifecycle_date_millis" : 1619644827810,
      "age" : "4.73d",
      "phase" : "hot",
      "phase_time" : "2021-05-03T14:40:42.715Z",
      "phase_time_millis" : 1620052842715,
      "action" : "rollover",
      "action_time" : "2021-04-28T21:30:28.123Z",
      "action_time_millis" : 1619645428123,
      "step" : "ERROR",
      "step_time" : "2021-05-03T14:50:41.446Z",
      "step_time_millis" : 1620053441446,
      "failed_step" : "check-rollover-ready",
      "is_auto_retryable_error" : true,
      "failed_step_retry_count" : 335,
      "step_info" : {
        "type" : "security_exception",
        "reason" : "action [indices:admin/rollover] is unauthorized for API key id [B3BJM4gBw7Tb8aIbES1t] of user [formeradmin@domain.tld]",
        "stack_trace" : "ElasticsearchSecurityException[action [indices:admin/rollover] is unauthorized for API key id [B3BJM4gBw7Tb8aIbES1t] of user [formeradmin@domain.tld]]

A user can lose permissions for too many reasons to name here. If it happens and the user is associated with ILM policies, it can create failures in ILM that can be very difficult to recover from. If not noticed, an index failing rollover like in the _ilm/explain output above can create indices with very large shards and cluster imbalance.

associated ES server log

[instance-0008675309] policy [indexpolicy] for index [index-000061] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
org.elasticsearch.ElasticsearchSecurityException: action [indices:admin/rollover] is unauthorized for API key id [B3BJM4gBw7Tb8aIbES1t] of user [formeradmin@domain.tld]
	at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:34) ~[x-pack-core-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:596) ~[x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.access$300(AuthorizationService.java:94) ~[x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.handleFailure(AuthorizationService.java:647) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:633) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:603) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$authorizeIndexAction$4(RBACEngine.java:319) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexActionName(RBACEngine.java:332) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeIndexAction(RBACEngine.java:303) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:269) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:234) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:199) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:121) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$7(CompositeRolesStore.java:234) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildAndCacheRoleFromDescriptors(CompositeRolesStore.java:265) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$8(CompositeRolesStore.java:232) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.buildAndCacheRoleFromDescriptors(CompositeRolesStore.java:265) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.lambda$getRoles$9(CompositeRolesStore.java:231) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authc.ApiKeyService.getRoleForApiKey(ApiKeyService.java:373) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:224) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:127) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:115) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:201) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:173) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:323) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:385) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:396) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:156) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:156) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:108) [x-pack-security-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:83) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:399) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:160) [x-pack-core-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.ilm.LifecyclePolicySecurityClient.doExecute(LifecyclePolicySecurityClient.java:51) [x-pack-ilm-7.8.1.jar:7.8.1]
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:399) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1234) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.rolloverIndex(AbstractClient.java:1736) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:128) [x-pack-core-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:173) [x-pack-ilm-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:329) [x-pack-ilm-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:267) [x-pack-ilm-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183) [x-pack-core-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:211) [x-pack-core-7.8.1.jar:7.8.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]

Please strongly consider decoupling ILM policy execution from the user setting the policy. Could ILM actions be executed as a built-in system user?

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[trackanstian]

Our situation
We were having similar issues with ILM for a while, where the rollover would not perform due to the user did not have access to write to the index. This has now become a huge problem for us.

Our management users that we use to edit the ILM policy does not as default have access to the indexes that the ILM policy applies to. (Due to GDPR restrictions). When the policy is edited, it means that the ILM policy is owned by that user and is run by that user. Currently there is also no way of showing who owns the ILM policy in the Elastic API or in Kibana. The ILM policy then fails if the user does not have access to the index.

Our "workaround" for this issue is that we have a script that runs every hour, saving the ILM policy with another user. This user then have access to the indexes. This is a terrible solution, but we need to make sure that the ILM policies can execute as this is trivial for running our environment. And since there is no way of knowing who owns the ILM policy we have to use that soluation witl all the Policies.

This is the answer from Elastic support:
We’ve asked a couple of our Elastic teams about speaking more to why we do this rather than having a system user execute the policies, and the primary reasoning is security related. We don’t want users to be able to set ILM policies on indices/etc that they themselves don’t have access to. The same approach is applied to our ML datafeeds, rollups, watches, and transforms - all to prevent background tasks from being exploited and used as a means of gaining access to data not intended for the user.
As options to manage this, we usually recommend things like:
• Having users log in as a dedicated admin user
• Doing something like creating an ILM admin account, then using the run as feature (https://www.elastic.co/guide/en/elasticsearch/reference/current/run-as-privilege.html) to have users perform the updates
We realize these aren’t particularly elegant solutions but we consider them preferred to allowing privilege escalation, and not a bug. There are discussions going on around how to make situations like yours less impactful to the operation, but nothing tangible yet.

As I see, this is a badly implemented feature as it is right now. I understand that you want to prevent ILM beeing able to be used as privilege escalation, but a solution could be to implement a security role on who is allowed to edit/run ILM Policies, insted of the current solution. We should at least be able to see who owns the policy.

1+ to this feature, or something similar.

[dakrone]

We discussed this today and decided that this requires a bit more discussion and brainstorming before we can decide on a path going forward. We've added it to our roadmap to come up with a design for how this will interact with other pieces of the Elastic stack (like Fleet for instance).

[joshdover]

@dakrone I realize this was a while back, but I'm curious if there was ever any design work done here?

[dakrone]

@joshdover not yet, we haven't done any work towards this recently. Other requests have come in that pushed this out of our list of things we can work on currently.

[tylerperk]

We recognize the need to improve the experience around permissions for ILM policies, so we do have an item related to this on the data management roadmap. We can't currently know when it might be implemented as it is lower priority than other items we have planned.