Backup .security

[albertzaharovits]

At present backing up the .security is achieved using the common snapshot APIs. The problem is that the snapshot and restore actions require a superuser if the snapshot will be including the .security index. This is a problem because periodic snapshotting is a good practice but using a superuser for it is obscuring the benefits.

A superuser is a user having the "superuser" role.
Only a superuser is able to read or write the ".security" index, as well as list it (in wildcards).

The Security Index contains:

• Tokens - used to authn users without password; a token is shorter lived than a password
• Users - attributes and hashed password for users under the Native & Reserved Realm
• Role Mappings - rules that assign users (based on their attributes) to roles
• Roles - description of permissions for each role
• App Privileges - description of app privileges (outside of ES authz model, eg Kibana Spaces)

Right now, snapshotting is equivalent to reading the index because the user can add a repository and then restore it under a cluster of its own control. Likewise, restoring is equivalent to writing the index, hence the tight authorization enforcement when the ".security" index is included.

Snapshot and Restore are authorized as cluster actions by the manage privilege. This privilege is required irrespective of the indices being snapshotted. The manage cluster privilege is a very powerful privilege.

The proposed action plan will be based upon the existing snapshot and restore infra as opposed to using separate APIs. All the following changes concern only the snapshot operation. The restore operation should not be automated and is OK for it to require superuser. Moreover, this superuser should reside in the file realm because the .security has to be closed before restoring and therefore it cannot be used for this operation.

EDITED 16.04.19
EDITED 25.06.19

Proposed action plan

1. create read-only system role for snapshotting internal indices (no other actions allowed, including restore which will still require superuser privileges); the internal indices set remains to be defined but it will include .security it is .security, .security-6 and .security-7 (and potentially .security-tokens and .security-tokens-7) .
   This has been completed by merging:

• Permission for restricted indices #37577
• Remove implicit index monitor privilege #37774
• Create snapshot role #35820

2. fork ephemeral contents of the .security index to another security index sibling. This sibling should not be ordinarily snapshoted or restored (but should be optionally included). This index is called .snapshot-tokens and the work has been completed by merging and backporting:

• Security Tokens moved to a new separate index.

3. support client library encryption for at least one repository type . Encryption on the cloud client appears to be more promising than encryption on the repository side because the latter might not be compatible with incremental snapshotting.
   The client side encryption of the cloud client SDKs is patchy: some support it - AWS, others don't - GCS, and they offer various controls over the key format and where it can reside. Moreover key rotation looks like a taboo topic if you opt to store the encryption key outside the provider's "Vault Service". For this matter we've opted for a flexible solution, in which we do encryption via the Java API so that we can store the encryption key in the node's keystore file, but also the key handler could designate a key (eg address in the "Vault Service") that is meaningful only to the cloud library SDK.
   This is in-progress with the pre-requisite work of:

• Consistent Secure Settings because data nodes with different encryption keys would amount to hard to detect failure mode.
• Client Side Encrypted Snapshot Repositories
  In the last team's meeting we decided to remove the encrypted snapshot repo prerequisite of this feature's (Backup .security). Chief reason being that the main motivation for this feature now comes from users with encryption at rest and key management policies; these requirements are on a different level than we originally anticipated, so the complexity of this feature now outgrows the complexity of the Backup .security for which encryption is only a nice to have.

4. Document (and maybe enforce) the snapshotting of the .security index only to a repository with encryption capabilities (see previous point). Documentation of the complete snapshot-restore flow (creating the file based role and user, closing .security, etc...) is also required.

[elasticmachine]

Pinging @elastic/es-security

[albertzaharovits]

Gentlemen from @elastic/es-security,

Please help me to outline the user experience for the new workflow of snapshotting the .security index.

Abstract
The snapshot request can specify multi index expressions. This expression is resolved irrespective of the user's privileges: there are no index privileges for snapshots, this is a cluster action.
In particular snapshotting for "*" will go and snapshot .security as well.

Segregated Repositories
There are still variables to peg, but it is likely we will end up with segregated repositories some for "ordinary" indices and some for "system" indices (read .security) which will be encrypted. From a user perspective, with the good habit of snapshotting "*", he will probably be surprised when snapshotting does not work as expected. He will have to read the documentation 😨 . The documentation will describe how to set up a repository for the system indices, and more importantly how to craft the index expressions for the two snapshot calls (excluding ",-.security*" for "ordinary" indices and including ".security*" for "system" ones). There have to be two API calls since a request targets a single repository.

The Workflow
There are many alternative stories of this process. I will set forth two of them.

• When a user tries to snapshot .security to an un-encrypted repository the operation will fail from the onset. He recreates the index expression to not include .security and continues to use the snapshot API with the existing repository. He then sets up the encrypted repository. He crafts another snapshot request that only includes the .security and calls the same snapshot API but with the name of the encrypted repository he created previously. This is my preference.

• When a user tries to snapshot .security to an un-encrypted repository the operation will snapshot "normal" indices but complain and return an error if the expression included the .security index . For the .security index there is a separate API (not snapshot, although it is a very light wrapper over it). This API does not require the .security index name as a parameter, but it does require the name of an encrypted repo.

As I said there quite some variants. I am not saying we should decide right now, since we don't have all the building blocks in place, but please keep this problem in the back of your head and voice your opinion.

I prefer the first alternative. If you have other ideas I am curious to discuss them. We can spin an email thread if discussion becomes involved. Otherwise, if I hear no howling, I will proceed with the mindset that the first alternative is sensible, and I will try to keep this feed updated as we unravel the mysteries.

[tvernum]

My personal view is that we shouldn't be requiring encrypted repos for backing up .security.
Encouraging and supporting, but not requiring.

I think we try too hard to make security decisions for other people without knowing their particular environment and needs, and that just makes things hard for them.

Can you backup .security to NFS, even though it's not encrypted?
If so, then why not S3 (which might be an on-prem S3-like)
If not, then why are we forcing people to push their .security backups out to the cloud rather than keeping them in their datacentre? (because our plan implies that only cloud repos will support encryption).

[jkakavas]

I agree with @tvernum 's comment on the encryption requirement. If I remember the discussions in Dublin correctly, the reasoning behind marking the (client side) encryption of repositories as a prerequisite is that so that we support(and encourage it as Tim said) it, but not enforce it. I get where you're coming from and I agree with the notion of protecting the users but Tim nailed it with

I think we try too hard to make security decisions for other people without knowing their particular environment and needs, and that just makes things hard for them.

Pending the discussion on the * index expansion in requests we could make sure that the user (either a superuser or a user with Snapshot Only role ) has explicit read privileges over system indices and that should get us as far as we need from the perspective of protecting the user.