Update privilege model for API keys

[tvernum]

The API Key actions are under the cluster:admin/xpack/security namespace.
The only cluster privileges that allow access to those actions are manage_security and all, both of which grant far more access than is actually required to create an API Key.

There should be a specific privilege to allow a user to create/delete their own API keys.

[elasticmachine]

Pinging @elastic/es-security

[bizybot]

Cluster privileges for API keys:
manage_api_key (cluster:admin/xpack/security/api_key/*)
Users with this cluster privilege can:
- create, invalidate or retrieve API keys.
- act on any API keys
This privilege is in-line with our philosophy to have more restricted privileges rather than giving wide access if not required.

create_api_key (cluster:admin/xpack/security/api_key/create)
Users with this cluster privilege can:
- only create API key(s)
API keys are more like bearer tokens and we expect admins want to have more control on who can create API keys.

owner_manage_api_key (cluster:admin/xpack/security/api_key/invalidate/my if we go by Option 1 described below)
Users with this cluster privilege can:
- only invalidate API keys owned by them

Following is the summarization of what is the expected behavior for API keys:-

User ▼ API Key Action ►	Create API key	Invalidate API key	GET API key
User with all or manage_security or manage_api_key privilege	Allow	Allow	Allow
User with create_api_key privilege	Allow	Deny	Allowed only on owned API keys
User with owner_manage_api_key privilege	Deny	Allowed only on owned API keys	Allowed only on owned API keys
User with no privilege related to API keys	Deny	Deny	Allowed only on owned API keys
User authenticated via API keys with no privilege related to API keys	Deny	Deny	Allowed only on owned API keys

Implementation options:
Implementing this would be challenge as we are trying to restrict action to a set of documents.
We would want our authorization checks to happen before reaching the APIs.

Option I:
Another option would be to add new REST endpoint
GET /_security/api_key/my
DELETE /_security/api_key/my
This way we make the username/api-key-id compulsory when we call into the APIs thereby restricting the action to authenticated user's API keys.

Option II:
So when we say Allowed only on owned API keys, we need to fetch the API keys for the authenticated user to authorize the user action. For example, User who owns API key and the request comes to invalidate API key by Id, in that case we need to first retrieve the API key(s) owned by this user and then check if the API key id is indeed owned by this user before proceeding with the invalidate action.
This feels more on the lines of object level security and implementing that does not seem to be viable option, as the design may not do the justice to other scenarios that are more appropriate for OLS.

[albertzaharovits]

There should be a specific privilege to allow a user to create/delete their own API keys.

I would like to voice my opinion that we should go for a full-blown ConditionalClusterPrivilege rather than introducing the ownership model. The general problem with the ownership model is that it leads to credentials being shared between parties that want shared ownership of a pool of tokens. Regarding implementation difficulty, I believe they're on the same level.

[bizybot]

I would like to voice my opinion that we should go for a full-blown ConditionalClusterPrivilege rather than introducing the ownership model. The general problem with the ownership model is that it leads to credentials being shared between parties that want shared ownership of a pool of tokens. Regarding implementation difficulty, I believe they're on the same level.

Thank you @albertzaharovits for raising this. I agree to the concern raised, though for now we have following:

• administrator account with manage_api_key can manage any API keys, agreed that this is wide access and someone wanting to restrict access to a group of API keys would not be possible.
• ownership model allowing users who own the API keys to manage them.

I did think about using ConditionalClusterPrivilege (I think Tim introduced as foundation towards OLS) but was not sure of it as we would need to autowire components so we can do a dynamic query to security index to fetch documents based on the configuration. If I am not mistaken the example you have is allowing access to group of API keys created by different users and using those principal names in ConditionalClusterPrivilege when we create role. When the authorization check happens we would use the principal names from the ConditionalClusterPrivilege, fetch the documents from index and then check against the request.

There could be multiple ways to group API keys, for example, grouping of API keys by username, by user metadata : department, by user metadata : group, by user metadata : location etc. Once you look at the way we could group objects, I think of a grouping object construct with a grouping service which allows us to resolve things. This seems more generic service and requires lot of thinking before we consider this or any alternative approach.

@albertzaharovits @tvernum @jkakavas
If we all agree, we can work on this as pathway to OLS or we could take this as next step.
Appreciate your inputs, Thank you.

[albertzaharovits]

Apologies for not being clear @bizybot ! I'll try to explain myself better:

The ConditionalClusterPrivilege is just an enhanced cluster privilege with the added power-up that it can authorize based on the request object's contents. The ConditionalClusterPrivileges.ManageApplicationPrivileges is the sole example.

In the case of API keys I see there are actions for:

• creating an API key for the self user, binding it to an apiKeyName (user specified) and an automatic apiKeyId
• retrieving API keys by users, realm, apiKeyId, apiKeyName
• invalidating API keys by users, realm, apiKeyId, apiKeyName

From my perspective, the privilege under discussion - to create/retrieve/invalidate ones own API keys - makes sense and would be difficult to model it as a ConditionalClusterPrivilege, but it just divides the clients into two groups: USERS that can create/retrieve/invalidate API keys for themselves and ADMINS that can retrieve and invalidate by any criteria. Though, given the "limiting" roles for the Create API Key action the clients can achieve some multi-tenancy among the ADMIN users (the ones creating API keys). So to be clear, I'm not against this, but to me it sounds more complicated than necessary.

What I was thinking was to have a new type of ConditionalClusterPrivilege that narrows all creating, retriving and invalidating to a restricted apiKeyName space. For example:

{
  "global": {
    "api_key": {
      "manage": ["some/api_key/namespace/*"]
    }
  }
}

Would permit creating API keys only for this namespace. Retrieving and Invalidating would work only by the same apiKeyName namespace (no users, realm, apiKeyId permitted).

What do you think @bizybot @tvernum ? This again is not a substitute for the ownership permission (and that cannot be modeled with ConditionalClusterPrivilege) but I think is a more natural way of limiting the privileges of API key admins.

[tvernum]

I like your proposal @albertzaharovits, except that I don't think it really solves the problem we have before us right now.

As we stand, it is impossible for users to be given access to manage their own API keys, which is a problem because it's a necessary feature for the sorts of use cases we want to support with API keys. Hadoop jobs should be able to create a new key, and then invalidate it when finished. We could just say that the only way to do that is by authenticating with the key itself, but then cleaning up after failed jobs is hard, and it makes the whole feature way too cumbersome.

The namespace proposal doesn't easily help because it makes it extraordinarily difficult to satisfy the common case. There is no simple way to say you can create, get and invalidate your own keys. To do that we would need some sort of template so you could set the namespace to /users/{username}/*, and even then it feels like a clunky solution to the problem at hand, in order to support things we're not sure we need.

I'm not tied to any particular implementation, but I am keen to see us solve the problem we know we have, and move on to higher priority work.

[albertzaharovits]

Fair enough, if the ownership privilege is necessary for Hadoop jobs and the namespace solution is not suitable then I have no case, sorry for rambling besides the point. I am not aware of the use case constraints.

[albertzaharovits]

As promised, here's what I have as a proposal for API-Key APIs. I will describe the REST interface and then detail how the transport interface might look like.

• create API-Key (POST _security/api_key) stays the same. It creates API keys for the authenticated principal calling the API. API keys are created under a user supplied unique name.
• get API-Key (GET _security/api_key) works a bit differently. When an authenticated principal calls the endpoint without any parameters, a list of API-Key names (no secret ids) is returned. The names are of that principal only (self). To get the (secret) id of the API-Key given a name, call GET _security/api_key/{api_key_name} . There are parameters to this API, to specify the user and/or realm. These parameters are available only to the manage_api_key administrator. This API is not available if the caller is authenticated with an API key. That's because if a user creates several API keys (probably with limited roles), and handles them to services, it would undesirable for one service to be able to retrieve API Keys of other services using this API. We could probably make the API retrieve only that single API key, but it would complicate the interface for very little benefit.
• invalidate API-Key (DELETE _security/api_key) also works a bit differently. To invalidate an API-Key by name call DELETE _security/api_key/{api_key_name}. Invalidation works in the authenticated user's namespace only (cannot invalidate other users keys, unless you specify the user request parameter). Invalidation is not available if the principal is that of an API key (I'm unsure about it, I prefer to not complicate the interface without a clear reason). Invalidation also does not work by id. The user and realm request parameters (specified together), which allow to invalidate keys for other users, are available only to the manage_api_key administrator.

Here are a few examples:

• curl -X POST -u joe _security/api_key?name=key_name_1 -> creates and returns the key (a secret id) , named key_name_1 for the principal joe. (name parameter could be a request body parameter, it is a URL parameter to simplify the examples)
  curl -X POST -u joe _security/api_key?name=key_name_2
  curl -X POST -u sally _security/api_key?name=key_name_1
  curl -X POST -u sally _security/api_key?name=key_name_3
• curl -X GET -u joe _security/api_key -> returns key_name_1 and key_name_2 in the response body, without the secret ids (the concrete API keys)
  curl -X GET -u sally _security/api_key -> returns key_name_1 and key_name_3
  curl -X GET -u joe _security/api_key/key_name_1 -> returns the secret id (the concrete API) key for key_name_1
  curl -X GET -u joe _security/api_key/key_name_3 -> MISSING
  curl -X GET -u sally _security/api_key/key_name_3 -> returns the secret id (the concrete API) key for key_name_3
  curl -X GET -u joe _security/api_key?user=joe -> DENIED
  curl -X GET -u joe _security/api_key?user=sally -> DENIED
  curl -X GET -u joe _security/api_key?realm=whatever -> DENIED
  curl -X GET -u elastic _security/api_key?user=sally&realm=whatever -> names key_name_1 and key_name_3
  curl -X GET -u elastic _security/api_key?realm=whatever -> names key_name_1 and key_name_2 for user joe and key_name_1 and key_name_3 for user sally
  curl -X GET -u elastic _security/api_key -> EMPTY , elastic does not have any keys of its own
  curl -X GET -u elastic _security/api_key?user=* -> all API key names (key_name_1 and key_name_2 for user joe and key_name_1 and key_name_3 for user sally)
  curl -X GET -H "Authentication: Bearer xxxx" _security/api_key?... -> DENIED (any request for GET api key is denied)
• curl -X DELETE -u joe _security/api_key -> ERROR (missing parameter)
  curl -X DELETE -u joe _security/api_key/key_name_1 -> invalidation successful
  curl -X DELETE -u joe _security/api_key/key_name_1 -> MISSING
  curl -X DELETE -u sally _security/api_key/key_name_3 -> invalidation successful
  curl -X DELETE -u elastic _security/api_key/key_name1?user=sally&realm=whatever -> invalidation successful
  curl -X DELETE -u elastic _security/api_key/key_name1 -> ERROR (missing parameter)
  curl -X DELETE -H "Authentication: Bearer xxxx" _security/api_key?... -> DENIED (any request for DELETE api key is denied)

The main points:

• if you authenticate with the API key you are not permitted to manage any API key.
• API-Key ids are secret, so we avoid to parse them around. When creating the key the client has the name and the secret id. If it loses the public name it cannot recover it, it is useless anyway because it cannot use it for anything useful.
• user and realm request parameters are permitted only to the manage_api_key administrator. Not even if they are used redundantly.

The reason to have these limitations is to simplify the transport level interface. I believe these can achieve transport interfaces for GET and DELETE where the request always contains the user and optionally (frequently) the API key name. In many cases the user has to be inferred in the REST handler, but this is a simple read of the thread context (and not a has_privilege call). During the authorization we always compare the principal in the request with the current authentication context. Wildcards (or null values) are permitted for all parameters, but are authorized only for the manage_api_key administrator.

The grand idea is that we must pass the user in the transport request, in all circumstances, to be able to authorize with the current framework that can inspect the request but not filter the response.

This deviates quite a bit from the current interface, but I am not attached to it at all, so we could change it anyway to closely follow what we currently have, but I believe we definitely should build around always having the owner user of the key in the transport request, without recurring to any has_privilege call in the REST handler.

[bizybot]

Thank you @albertzaharovits for your proposal.

I went through your comment and I am not very convinced with the proposal.
Also, I see that in your proposal you consider the API key id to be a secret but it is similar to username in case of native users. The actual key is similar to the password which cannot be retrieved by any APIs but only available on the creation of API key. I am not sure if this is the reason that your proposal is proposed the way to get the API key id using the key name and then do stuff.

API-Key ids are secret, so we avoid to parse them around. When creating the key the client has the name and the secret id. If it loses the public name it cannot recover it, it is useless anyway because it cannot use it for anything useful.

When we create an API key api_key is similar to a password that needs to be protected but I am not sure if there is any advantage with API key-id being a secret and complicating the workflow. I am considering the API key id analogous to the user id or REST resource id.

{
  "id":"VuaCfGcBCdbkQm-e5aOx", 
  "name":"my-api-key",
  "expiration":1544068612110, 
  "api_key":"ui2lp2axTNmsyakw9tvNnw" 
}

I will start with the targeted use case before diving into the API details:-
a) As an Administrator, I would like to manage API keys of any user and/or any realm.
(manage_api_key privilege)

GET OR DELETE /_security/api_key:
The administrator is likely to use parameters (user or realm or both) when using any of the APIs.
In a specific case when it wants to use API key-id or name to invalidate an API key then that is allowed as well.
(With the current behavior I think we do not have any problem here)

b) As a user, I would like to create API key(s) and be able to manage owned API keys
(manage_own_api_key)

GET OR DELETE /_security/api_key:
A user is most likely to use (id or name) to retrieve or invalidate API keys and not specify user or realm parameters.
In the case where it chooses to use the user or realm parameters then we would check if the currently authenticated user has the same user and realm as the authenticated user's. If not then we will deny access with a proper error message.

c) As an agent (say device agents of a fleet) using an API key for authentication, I would like to retrieve my (API key) information but not invalidate my API key or any other API keys. (The API key does not carry information which the agent would want to retrieve later, for example, expiry date)
(Since the API key is used for authentication, this has permission to retrieve self, other API invocations will fail because of authorization checks)

Given these scenarios and the API behavior do we think that we need to modify the behavior of the API? I would first consider if there is anything in above that causes confusing behavior from the API usage perspective and then consider the alternatives. Thank you.