Metadata for API keys

[bytebilly]

Description

We are considering to extend Elasticsearch API keys in order to support internal requirements by other teams, and avoid building new features that would increase complexity and maintenance costs.

The current implementation is a good starting point, but hits its limits in complex scenarios where those keys are used in a more sophisticated authentication flow.
At the moment, the solution is to build an external logic around keys, or reimplement an independent similar feature.

Proposal

One thing that could be very useful is to allow custom arbitrary metadata to be attached to an API key during the creation action. This is totally transparent to Elasticsearch, that has no knowledge of the meaning (and format) of the metadata, and doesn't perform any action on it.

An example of metadata could be the scope of the key, a description, or a cryptographic signature.

This implies that metadata can be passed to the Create API key call, stored as an attribute of the newly generated key, and then returned by the Get API key.

Each user can define the metadata, and build some logic around it.

Related Kibana issue: elastic/kibana#93820

[elasticmachine]

Pinging @elastic/es-security (:Security/Authentication)

[bytebilly]

@arisonl @roncohen @tvernum as we discussed in previous conversations, this could be a very turning point to simplify the authentication and authorization flow for Ingest, where we actually use a separate service.

I'd like to keep it very generic, so that Elasticsearch has no knowledge of the metadata content, and it can be used in very different scenarios.

This is probably the only requirement on the Elasticsearch side to make API keys suitable for Ingest needs, but I'll continue to investigate about that.

[sorenlouv]

allow custom arbitrary metadata to be attached to an API key during the creation action

+1. This is needed for elastic/kibana#77966 (currently blocked until we can do something like that).

Use case
For APM we want to make it easier for users to create API keys that are used between an APM agent (eg the java agent) and the APM Server.
Currently we can create API keys, but have no way of retrieving them again since we cannot annotate API keys with apm or similar. We might also want to annotate the API key with additional metadata like a description, the environment etc.

[tvernum]

@sqren

I would have expected that all of those API keys would be owned by the same user (apm_system ?) and you would be able to retrieve them that way. Is that not the case?

[graphaelli]

Jumping in as @sqren is on vacation right now. Can you clarify whether the question is; are all API keys owned by a single user? Following on, if they were, would that user only have API keys used for agent/server auth?

Currently, API keys used for this purpose are not owned by a single user, they're not even created via kibana at the moment (see elastic/kibana#77966 for details).

[tvernum]

Shouldn't they be owned by a single user? Do you intentionally want APM API Keys to be owned and managed by individuals?

[simitt]

Right now there is no UI for creating API keys, non-cloud users can use a CLI tool built into the APM Server, cloud users need to follow the docs for creating the API key manually.
The CLI tool is beta and creates the API Keys via the provided user. The name of the apm_system user is a bit misleading as it has been created and used for writing monitoring data, aligned with the beats_system user. I don't think that user would necessarily be the right one to use here, at least we would need to give it additional privileges and update the docs accordingly.
We honestly also didn't consider creating/using a built in APM user for this. Which advantages would you see for creating this by a system user rather than an individual user?