From 87cea12b540c5c93e9cff17556935ead09395b51 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Thu, 6 Jun 2019 19:44:05 +0300 Subject: [PATCH 01/42] Sketchy --- .../backup-security-configuration.asciidoc | 142 ++++++++++++++++++ 1 file changed, 142 insertions(+) create mode 100644 x-pack/docs/en/security/backup-security-configuration.asciidoc diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc new file mode 100644 index 0000000000000..d1f4433627ea2 --- /dev/null +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -0,0 +1,142 @@ +[role="xpack"] +[[backup-restore-security-configuration]] +=== Security Configuration + +Security is configured from files and by various API endpoints. All Security +settings are namespaced under `xpack.security.*`. + +The configuration files reside on every node but are usually identical across +nodes. They are formatted as YAML text files. Also on every node, there is a +secure store file, named `elasticsearch.keystore`, which is binary formatted, +and it is used only for sensitive settings such as service passwords. Some setting +values may be paths pointing to other files, such as the keys and certificates to +configure TLS. Reassuringly, all these files are stored inside the +`$ES_PATH_CONF` directory, so backing-up the whole folder is sure to capture +all of Security's file based configuration. + +The API calls act on configuration that is stored in the global cluster +metadata as well as inside a dedicated {es} index. This index is named +`.security-6` in the 6.x versions and `.security-7` in the 7.x releases, and it +is always pointed to by the `.security` alias. This contains the data which is +*not* part of an ordinary configuration backup. This data describes: + +* definition of users in the native realm (including hashed passwords) +* role definitions +* role mappings +* application privileges +* API keys + +The format is only compatible across the same major release version. + +The `elasticsearch.yml` and `elasticsearch.keystore` files in `$ES_PATH_CONF`, +as well as the global cluster metadata contain settings for all the cluster's +features, not only for Security. It is easier to backup and restore all these +configurations as a whole, and also, non-Security configurations need backup too! +Cherry-picking only Security configurations complicates the procedure a bit, but +it's definitely a viable option. All the other +{es-repo-dir}/security/reference/files.asciidoc[configuration files] and the +`.security` index contain only Security configuration data. + +=== Backup File Based Configuration + +Changes to the configuration files usually require a node restart for the +changes to take effect, so this configuration method is reserved for settings +that change rarely (or are required for cluster formation, or require file +system privileges for modifications). In a typical deployment, once a desired +configuration is reached, subsequent changes will be infrequent. + +The simplest way to backup file based configuration is to simply copy the +`$ES_PATH_CONF` directory. And given the infrequent modifications, you can do +this after every change. For the YAML text files you also have the option of +storing diffs against a baseline, or checking them into `git`, to make the +changes more easily reviewable by the {es} administrator. The binary or encoded +files, such as the private keys or the secure store file are extra sensitive; +they require restricted read access. This is the identical procedure as for a +complete backup. Remember that in a typical {es} deployment all files are +indentical on every node. + +=== Backup API Based Configuration + +API based configuration backup uses the common snapshot functionality [link]. +The snapshot API saves the settings, the type mapping and the contents of any +index, to a location of a cloud storage service or on a shared network file +system. The snapshot can also optionally capture the persistent settings from +the global cluster metadata. + +To use the snapshot API to backup `.security` first create a repository [link]. +Creating a repository requires the elevated `manage` cluster privilege. +Afterwards, all snapshots can be created inside that repository by a user with +the limited `snapshot_user` role. This role grants privileges to create +snapshots in any existing repositories. It also grants privileges to list and +read settings for all indices (but no data). + +[example creating the user] +[example doing the snapshot] + +A few Security settings are stored in the global cluster metadata as persistent +settings. These are included in the snapshot, but for facilitating change +reviews as well as restore convenience, we can pull them out to stand on their +own file. + +curl -X GET -u elastic:password "localhost:9200/_cluster/settings?pretty&flat_settings" | jq '.persistent | with_entries(select(.key|startswith("xpack.security")))' + +You can store the output of this together with the other configuration files. + +NOTE: Transient settings are not considered for backup. + +=== Backup Repository Access + +Simply exposing the contents of the `.security` index, by having a third party +read the snapshot, does not equate to a compromised cluster security. This is +because an attacker still needs to brute force cryptographic hashes before it +can impersonate users or use API keys. But every document in the security index +is sensitive to changes, meaning that any modification has the potential to +compromise the integrity of all the data in the cluster. Therefore it is +recommended to: + +* snapshot `.security` in a dedicated repository, to which read and write +access is strictly restricted and audited +* if there are indications that the snapshot has been read, proceed to changing +the passwords of the users in the native realm as well as revoking API keys +* never restore a snapshot for which there are indications that it has been +tampered with; there is currently no option for the restore process to detect +malicious tampering + +The same recommendations are available for the storing of backups for +configuration files. However, special caution must be employed when storing +the `elasticsearch.keystore` and private key file backups, because currently +the secure store is *not* password protected and any read access can compromise +passwords of external systems and private keys. + +=== Restore Security Configuration + +The first part of restoring a Security configuration backup is restoring the +`.security` index. You can restore a snapshot of the `.security` index only if +it has been created in the same major release version. The last minor version of +every major release can convert and read both versions of the index. +Log in to one of the node hosts, navigate to {es} installation directory, and +follow these steps: + + * make sure the repository holding the `.security` snapshot is installed + * add a new user with the `superuser` role to the file based realm + bin/elasticsearch-users useradd jacknich -p password -r superuser + * using the previously create user, delete the existing `.security-6` or +`.security-7` index + curl -u jacknich:password -X DELETE "localhost:9200/.security-7" + + + NOTE: after this step authentication will not work so all API calls will fail + * using the same user, restore the `.security` index from the snapshot + curl -u elastic:password -X POST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -H 'Content-Type: application/json' -d' + { + "indices": ".security-7", + "include_global_state": true + } + ' + + + NOTE: restoring the global state is optional, but it will help make sure the + +The next part is really common to every other backup restore; copy the backed-up +configuration overwritting the contents of `$ES_PATH_CONF` and restart the node. + +You can now cherry-pick and apply the persistent secure settings: +curl -X PUT _settings From f0f32a3479c99418d1741bf881d43b6a91f089ef Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Fri, 7 Jun 2019 02:35:27 +0300 Subject: [PATCH 02/42] Show --- .../backup-security-configuration.asciidoc | 180 ++++++++++++------ .../docs/en/security/configuring-es.asciidoc | 5 +- 2 files changed, 125 insertions(+), 60 deletions(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index d1f4433627ea2..c5087735cb1d4 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -1,18 +1,24 @@ [role="xpack"] [[backup-restore-security-configuration]] -=== Security Configuration +=== Backup and Restore Security Configuration -Security is configured from files and by various API endpoints. All Security -settings are namespaced under `xpack.security.*`. +The {es} {security-features} are configured by files and by various API +endpoints. All these settings are namespaced under `xpack.security.*`. These +are ordinarily included in regular backups but the backup is not complete until +a special index, which holds most of the non-setting configuration, is also +included. -The configuration files reside on every node but are usually identical across +[float] +==== Security Configuration Explained + +The configuration files reside on every node and are usually identical across nodes. They are formatted as YAML text files. Also on every node, there is a secure store file, named `elasticsearch.keystore`, which is binary formatted, and it is used only for sensitive settings such as service passwords. Some setting values may be paths pointing to other files, such as the keys and certificates to configure TLS. Reassuringly, all these files are stored inside the -`$ES_PATH_CONF` directory, so backing-up the whole folder is sure to capture -all of Security's file based configuration. +`$ES_PATH_CONF` directory, so backing-up the whole directory is sure to capture +all of the file based configuration of {security-features}. The API calls act on configuration that is stored in the global cluster metadata as well as inside a dedicated {es} index. This index is named @@ -26,65 +32,98 @@ is always pointed to by the `.security` alias. This contains the data which is * application privileges * API keys -The format is only compatible across the same major release version. +Be advised that the index format is only compatible over a single major version. The `elasticsearch.yml` and `elasticsearch.keystore` files in `$ES_PATH_CONF`, as well as the global cluster metadata contain settings for all the cluster's -features, not only for Security. It is easier to backup and restore all these -configurations as a whole, and also, non-Security configurations need backup too! -Cherry-picking only Security configurations complicates the procedure a bit, but -it's definitely a viable option. All the other -{es-repo-dir}/security/reference/files.asciidoc[configuration files] and the -`.security` index contain only Security configuration data. - -=== Backup File Based Configuration - -Changes to the configuration files usually require a node restart for the -changes to take effect, so this configuration method is reserved for settings +features, not only the {security-features}. It is easier to backup and restore +all these configurations as a whole. Cherry-picking only {security-features} +configurations complicates the procedure a bit, without many benefits because +, after all, every configuration requires a backup! All the other <> and the `.security` index contain exclusively {security-features} +configuration data. + +[float] +==== Backup of File Configuration + +The simplest way to backup file based configuration is a trivial copy of the +`$ES_PATH_CONF` directory. Overall, this is the identical procedure as for a +complete backup. Also, remember that in a typical {es} deployment all files +are identical on every node. + +Because changes to the configuration files usually require a node restart for +the changes to take effect, this configuration method is reserved for settings that change rarely (or are required for cluster formation, or require file system privileges for modifications). In a typical deployment, once a desired -configuration is reached, subsequent changes will be infrequent. - -The simplest way to backup file based configuration is to simply copy the -`$ES_PATH_CONF` directory. And given the infrequent modifications, you can do -this after every change. For the YAML text files you also have the option of -storing diffs against a baseline, or checking them into `git`, to make the -changes more easily reviewable by the {es} administrator. The binary or encoded -files, such as the private keys or the secure store file are extra sensitive; -they require restricted read access. This is the identical procedure as for a -complete backup. Remember that in a typical {es} deployment all files are -indentical on every node. - -=== Backup API Based Configuration - -API based configuration backup uses the common snapshot functionality [link]. -The snapshot API saves the settings, the type mapping and the contents of any -index, to a location of a cloud storage service or on a shared network file -system. The snapshot can also optionally capture the persistent settings from -the global cluster metadata. - -To use the snapshot API to backup `.security` first create a repository [link]. -Creating a repository requires the elevated `manage` cluster privilege. -Afterwards, all snapshots can be created inside that repository by a user with +configuration is attained, subsequent changes will be infrequent. + +Given this, it is recommended to backup after every modification. For the YAML +text files you also have the option of storing diffs against a baseline, or +checking them into `git`, therefore making the changes more easi to review by +the {es} administrator. The binary and encoded files, such as the private keys +or the secure store file are extra sensitive; they require restricted read +access. + +[float] +==== Backup of API Based Configuration + +The backup of API based configuration uses the <>. The snapshot API saves the settings, the type +mapping and the contents of any index, to a location of a cloud storage service +or on a shared network file system. The snapshot operation can also optionally +capture the persistent settings from the global cluster metadata. + + . To use the snapshot API to backup `.security` first create a repository. +Creating it requires the elevated `manage` cluster privilege, yet it's a +relatively infrequent operation. The elevated privileges are required to prevent +regular users from exfiltrating data to locations of their choosing. + . Afterwards, all snapshots can be created inside that repository by a user with the limited `snapshot_user` role. This role grants privileges to create snapshots in any existing repositories. It also grants privileges to list and read settings for all indices (but no data). - -[example creating the user] -[example doing the snapshot] - -A few Security settings are stored in the global cluster metadata as persistent -settings. These are included in the snapshot, but for facilitating change -reviews as well as restore convenience, we can pull them out to stand on their -own file. - -curl -X GET -u elastic:password "localhost:9200/_cluster/settings?pretty&flat_settings" | jq '.persistent | with_entries(select(.key|startswith("xpack.security")))' ++ +-- +The following example creates a new user `snapshot_user` in the native realm: + +[source,js] +-------------------------------------------------- +POST /_security/user/snapshot_user +{ + "password" : "secret", + "roles" : [ "snapshot_user" ] +} +-------------------------------------------------- + +And the next one exemplifies the create snapshot operation, inside the +`my_backup` repository: + +[source,js] +-------------------------------------------------- +PUT /_snapshot/my_backup/snapshot_1 +{ + "indices": ".security", + "include_global_state": true <1> +} +-------------------------------------------------- + +<1> A few {security-features} settings are stored in the global cluster metadata as +persistent settings. These are included in the snapshot by, but for facilitating +change reviews as well as restore convenience, we can pull them out to stand in +their own file. + +[source,shell] +-------------------------------------------------- +curl -X GET -u elastic "localhost:9200/_cluster/settings?pretty&flat_settings" | jq '.persistent | with_entries(select(.key|startswith("xpack.security")))' +-------------------------------------------------- +// NOTCONSOLE You can store the output of this together with the other configuration files. NOTE: Transient settings are not considered for backup. +-- -=== Backup Repository Access +[float] +==== Backup Repository Access Simply exposing the contents of the `.security` index, by having a third party read the snapshot, does not equate to a compromised cluster security. This is @@ -108,7 +147,8 @@ the `elasticsearch.keystore` and private key file backups, because currently the secure store is *not* password protected and any read access can compromise passwords of external systems and private keys. -=== Restore Security Configuration +[float] +==== Restore Security Configuration The first part of restoring a Security configuration backup is restoring the `.security` index. You can restore a snapshot of the `.security` index only if @@ -117,21 +157,45 @@ every major release can convert and read both versions of the index. Log in to one of the node hosts, navigate to {es} installation directory, and follow these steps: - * make sure the repository holding the `.security` snapshot is installed - * add a new user with the `superuser` role to the file based realm + . make sure the repository holding the `.security` snapshot is installed + + +-- +[source,shell] +-------------------------------------------------- +GET /_snapshot/my_backup +-------------------------------------------------- +-- + . add a new user with the `superuser` role to the file based realm ++ +-- +[source,shell] +-------------------------------------------------- bin/elasticsearch-users useradd jacknich -p password -r superuser - * using the previously create user, delete the existing `.security-6` or +-------------------------------------------------- +-- + . using the previously create user, delete the existing `.security-6` or `.security-7` index - curl -u jacknich:password -X DELETE "localhost:9200/.security-7" ++ +-- +[source,shell] +-------------------------------------------------- + curl -u jacknich-X DELETE "localhost:9200/.security-7" +-------------------------------------------------- + NOTE: after this step authentication will not work so all API calls will fail +-- * using the same user, restore the `.security` index from the snapshot - curl -u elastic:password -X POST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -H 'Content-Type: application/json' -d' ++ +-- +[source,shell] +-------------------------------------------------- + curl -u jacknich -X POST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -H 'Content-Type: application/json' -d' { "indices": ".security-7", "include_global_state": true } ' +-------------------------------------------------- + NOTE: restoring the global state is optional, but it will help make sure the diff --git a/x-pack/docs/en/security/configuring-es.asciidoc b/x-pack/docs/en/security/configuring-es.asciidoc index fdc49ef21e213..55f0a6fe164e3 100644 --- a/x-pack/docs/en/security/configuring-es.asciidoc +++ b/x-pack/docs/en/security/configuring-es.asciidoc @@ -162,7 +162,8 @@ include::authentication/configuring-saml-realm.asciidoc[] :edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/security/authentication/configuring-kerberos-realm.asciidoc include::authentication/configuring-kerberos-realm.asciidoc[] -:edit_url: -include::fips-140-compliance.asciidoc[] +:edit_url: +include::backup-security-configuration.asciidoc[] include::{es-repo-dir}/security/reference/files.asciidoc[] +include::fips-140-compliance.asciidoc[] From 72a899207f99ceab766bd53d2e1002e1e37426b3 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Fri, 7 Jun 2019 02:45:34 +0300 Subject: [PATCH 03/42] Show --- .../backup-security-configuration.asciidoc | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index c5087735cb1d4..b773cb42c7e49 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -181,10 +181,10 @@ GET /_snapshot/my_backup -------------------------------------------------- curl -u jacknich-X DELETE "localhost:9200/.security-7" -------------------------------------------------- - + - NOTE: after this step authentication will not work so all API calls will fail + +NOTE: after this step authentication will not work so all API calls will fail -- - * using the same user, restore the `.security` index from the snapshot + . using the same user, restore the `.security` index from the snapshot + -- [source,shell] @@ -196,11 +196,12 @@ GET /_snapshot/my_backup } ' -------------------------------------------------- - + - NOTE: restoring the global state is optional, but it will help make sure the + +NOTE: restoring the global state is optional, but it will help make sure the +-- The next part is really common to every other backup restore; copy the backed-up configuration overwritting the contents of `$ES_PATH_CONF` and restart the node. -You can now cherry-pick and apply the persistent secure settings: -curl -X PUT _settings +Lastly, you can now cherry-pick and <> from earlier. From ef82389b44642f75b87c2488e9b4ff12534d21f6 Mon Sep 17 00:00:00 2001 From: lcawl Date: Wed, 12 Jun 2019 12:03:26 -0700 Subject: [PATCH 04/42] [DOCS] Changes titles to sentence case --- .../backup-security-configuration.asciidoc | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index b773cb42c7e49..87b3e202d979d 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -1,6 +1,6 @@ [role="xpack"] [[backup-restore-security-configuration]] -=== Backup and Restore Security Configuration +=== Backing up and restoring security configuration The {es} {security-features} are configured by files and by various API endpoints. All these settings are namespaced under `xpack.security.*`. These @@ -9,7 +9,7 @@ a special index, which holds most of the non-setting configuration, is also included. [float] -==== Security Configuration Explained +==== Security configuration explained The configuration files reside on every node and are usually identical across nodes. They are formatted as YAML text files. Also on every node, there is a @@ -39,12 +39,12 @@ as well as the global cluster metadata contain settings for all the cluster's features, not only the {security-features}. It is easier to backup and restore all these configurations as a whole. Cherry-picking only {security-features} configurations complicates the procedure a bit, without many benefits because -, after all, every configuration requires a backup! All the other <> and the `.security` index contain exclusively {security-features} -configuration data. +, after all, every configuration requires a backup! All the other +<> and the `.security` index contain +exclusively {security-features} configuration data. [float] -==== Backup of File Configuration +==== Backup of file configuration The simplest way to backup file based configuration is a trivial copy of the `$ES_PATH_CONF` directory. Overall, this is the identical procedure as for a @@ -65,7 +65,7 @@ or the secure store file are extra sensitive; they require restricted read access. [float] -==== Backup of API Based Configuration +==== Backup of API-based configuration The backup of API based configuration uses the <>. The snapshot API saves the settings, the type @@ -123,7 +123,7 @@ NOTE: Transient settings are not considered for backup. -- [float] -==== Backup Repository Access +==== Backup repository access Simply exposing the contents of the `.security` index, by having a third party read the snapshot, does not equate to a compromised cluster security. This is @@ -148,7 +148,7 @@ the secure store is *not* password protected and any read access can compromise passwords of external systems and private keys. [float] -==== Restore Security Configuration +==== Restore security configuration The first part of restoring a Security configuration backup is restoring the `.security` index. You can restore a snapshot of the `.security` index only if @@ -197,11 +197,11 @@ NOTE: after this step authentication will not work so all API calls will fail ' -------------------------------------------------- -NOTE: restoring the global state is optional, but it will help make sure the +NOTE: Restoring the global state is optional, but it will help make sure the -- The next part is really common to every other backup restore; copy the backed-up -configuration overwritting the contents of `$ES_PATH_CONF` and restart the node. +configuration overwriting the contents of `$ES_PATH_CONF` and restart the node. Lastly, you can now cherry-pick and <> from earlier. From 288a1b0e1bb8ff8c71011560b7dd649e24b61bd8 Mon Sep 17 00:00:00 2001 From: lcawl Date: Wed, 12 Jun 2019 13:31:53 -0700 Subject: [PATCH 05/42] [DOCS] Edits security backup steps --- .../backup-security-configuration.asciidoc | 140 ++++++++++-------- 1 file changed, 80 insertions(+), 60 deletions(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index 87b3e202d979d..5f569ba39c485 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -3,30 +3,32 @@ === Backing up and restoring security configuration The {es} {security-features} are configured by files and by various API -endpoints. All these settings are namespaced under `xpack.security.*`. These -are ordinarily included in regular backups but the backup is not complete until -a special index, which holds most of the non-setting configuration, is also -included. +endpoints. The security settings, which are prefixed with `xpack.security`, are +ordinarily included in regular backups. However, the backup is not complete +until a special index, which holds most of the non-setting configuration, is +also included. [float] -==== Security configuration explained +[[backup-restore-security-location]] +==== Location of security configuration details The configuration files reside on every node and are usually identical across -nodes. They are formatted as YAML text files. Also on every node, there is a -secure store file, named `elasticsearch.keystore`, which is binary formatted, -and it is used only for sensitive settings such as service passwords. Some setting -values may be paths pointing to other files, such as the keys and certificates to -configure TLS. Reassuringly, all these files are stored inside the -`$ES_PATH_CONF` directory, so backing-up the whole directory is sure to capture -all of the file based configuration of {security-features}. - -The API calls act on configuration that is stored in the global cluster -metadata as well as inside a dedicated {es} index. This index is named -`.security-6` in the 6.x versions and `.security-7` in the 7.x releases, and it -is always pointed to by the `.security` alias. This contains the data which is -*not* part of an ordinary configuration backup. This data describes: - -* definition of users in the native realm (including hashed passwords) +nodes. They are formatted as YAML text files. Also, on every node there is a +secure store file named `elasticsearch.keystore`, which is binary formatted. +This file is used only for sensitive settings such as service passwords. Some +setting values are paths pointing to other files, such as the keys and +certificates to configure TLS. Reassuringly, all these files are stored inside +the `$ES_PATH_CONF` directory, so when you back up the whole directory, you +capture all of the file-based configuration for the {security-features}. + +The API calls act on configuration information that is stored in the global +cluster metadata as well as inside a dedicated {es} index. This index is named +`.security-6` in the 6.x versions and `.security-7` in the 7.x releases. The +`.security` alias always points to the appropriate index. This index contains +the data which is *not* part of an ordinary configuration backup. This data +describes: + +* the definition of users in the native realm (including hashed passwords) * role definitions * role mappings * application privileges @@ -34,53 +36,66 @@ is always pointed to by the `.security` alias. This contains the data which is Be advised that the index format is only compatible over a single major version. -The `elasticsearch.yml` and `elasticsearch.keystore` files in `$ES_PATH_CONF`, -as well as the global cluster metadata contain settings for all the cluster's -features, not only the {security-features}. It is easier to backup and restore -all these configurations as a whole. Cherry-picking only {security-features} -configurations complicates the procedure a bit, without many benefits because -, after all, every configuration requires a backup! All the other -<> and the `.security` index contain -exclusively {security-features} configuration data. +TIP: The `elasticsearch.yml` and `elasticsearch.keystore` files and the global +cluster metadata contain settings for all the {stack} features. The +`.security` index and all the other +<> contain configuration data +exclusively for the {security-features}. It is easiest to backup and restore all +the configuration information as a whole, rather than trying to cherry-pick only +the security details. [float] -==== Backup of file configuration - -The simplest way to backup file based configuration is a trivial copy of the -`$ES_PATH_CONF` directory. Overall, this is the identical procedure as for a -complete backup. Also, remember that in a typical {es} deployment all files -are identical on every node. - -Because changes to the configuration files usually require a node restart for -the changes to take effect, this configuration method is reserved for settings -that change rarely (or are required for cluster formation, or require file -system privileges for modifications). In a typical deployment, once a desired -configuration is attained, subsequent changes will be infrequent. - -Given this, it is recommended to backup after every modification. For the YAML -text files you also have the option of storing diffs against a baseline, or -checking them into `git`, therefore making the changes more easi to review by -the {es} administrator. The binary and encoded files, such as the private keys -or the secure store file are extra sensitive; they require restricted read -access. +[[backup-restore-security-file]] +==== Backing up configuration files + +The simplest way to backup configuration files is to copy the `$ES_PATH_CONF` +directory. Overall, this procedure is identical to a complete backup. + +TIP: In a typical {es} deployment, all the configuration files are identical on +each node. + +It is recommended that you back up the configuration files every time they are +modified. In a typical deployment, after you achieve a desired configuration, +you update these files infrequently. If you change the contents of the +configuration files, the changes usually don't take effect until you restart +the node. Therefore, configuration files should contain only the settings that +change rarely or are required for cluster formation or require file system +privileges. + +For the YAML text files, you also have the option of storing diffs against a +baseline or checking them into `git`. This method makes it easier for the {es} +administrator to review the changes. + +The binary and encoded files, such as the private keys and the secure store +file, are extra sensitive; they require restricted read access. [float] -==== Backup of API-based configuration +==== Backing up API-based configuration -The backup of API based configuration uses the <>. The snapshot API saves the settings, the type -mapping and the contents of any index, to a location of a cloud storage service -or on a shared network file system. The snapshot operation can also optionally -capture the persistent settings from the global cluster metadata. +To back up API-based configuration, you use the +<>. The snapshot API saves the +settings, the type mapping, and the contents of any index to a location on a +cloud storage service or on a shared network file system. The snapshot operation +can also optionally capture the persistent settings from the global cluster +metadata. - . To use the snapshot API to backup `.security` first create a repository. -Creating it requires the elevated `manage` cluster privilege, yet it's a +. Create a repository that you can use to backup the `.security` index. ++ +-- +You must have the elevated `manage` cluster privilege to create it, yet it's a relatively infrequent operation. The elevated privileges are required to prevent regular users from exfiltrating data to locations of their choosing. - . Afterwards, all snapshots can be created inside that repository by a user with -the limited `snapshot_user` role. This role grants privileges to create + +Afterwards, users with the limited `snapshot_user` built-in role can create +snapshots inside that repository. This role grants privileges to create snapshots in any existing repositories. It also grants privileges to list and -read settings for all indices (but no data). +read settings for all indices (but not to read data). + +For more information, see {stack-ov}/security-privileges.html[Security privileges] +and {stack-ov}/built-in-roles.html[Built-in roles]. +-- + +. Create users that have the `snapshot_user` built-in role. + -- The following example creates a new user `snapshot_user` in the native realm: @@ -93,8 +108,12 @@ POST /_security/user/snapshot_user "roles" : [ "snapshot_user" ] } -------------------------------------------------- +-- -And the next one exemplifies the create snapshot operation, inside the +. Create a snapshot. ++ +-- +The following example exemplifies the create snapshot operation, inside the `my_backup` repository: [source,js] @@ -106,7 +125,7 @@ PUT /_snapshot/my_backup/snapshot_1 } -------------------------------------------------- -<1> A few {security-features} settings are stored in the global cluster metadata as +<1> A few security settings are stored in the global cluster metadata as persistent settings. These are included in the snapshot by, but for facilitating change reviews as well as restore convenience, we can pull them out to stand in their own file. @@ -120,6 +139,7 @@ curl -X GET -u elastic "localhost:9200/_cluster/settings?pretty&flat_settings" | You can store the output of this together with the other configuration files. NOTE: Transient settings are not considered for backup. + -- [float] From 7eb95a88c48a6f2554dcd0a7a83408cac559624c Mon Sep 17 00:00:00 2001 From: lcawl Date: Wed, 12 Jun 2019 14:49:28 -0700 Subject: [PATCH 06/42] [DOCS] Adds anchors --- .../backup-security-configuration.asciidoc | 31 ++++++++++++------- 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index 5f569ba39c485..65e565ce1bd3f 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -9,7 +9,7 @@ until a special index, which holds most of the non-setting configuration, is also included. [float] -[[backup-restore-security-location]] +[[backup-security-location]] ==== Location of security configuration details The configuration files reside on every node and are usually identical across @@ -45,7 +45,7 @@ the configuration information as a whole, rather than trying to cherry-pick only the security details. [float] -[[backup-restore-security-file]] +[[backup-security-files]] ==== Backing up configuration files The simplest way to backup configuration files is to copy the `$ES_PATH_CONF` @@ -70,6 +70,7 @@ The binary and encoded files, such as the private keys and the secure store file, are extra sensitive; they require restricted read access. [float] +[[backup-security-snapshots]] ==== Backing up API-based configuration To back up API-based configuration, you use the @@ -143,7 +144,8 @@ NOTE: Transient settings are not considered for backup. -- [float] -==== Backup repository access +[[backup-security-repos]] +===== Controlling access to the backup repository Simply exposing the contents of the `.security` index, by having a third party read the snapshot, does not equate to a compromised cluster security. This is @@ -168,7 +170,8 @@ the secure store is *not* password protected and any read access can compromise passwords of external systems and private keys. [float] -==== Restore security configuration +[[restore-security-configuration]] +==== Restoring security configuration The first part of restoring a Security configuration backup is restoring the `.security` index. You can restore a snapshot of the `.security` index only if @@ -177,34 +180,37 @@ every major release can convert and read both versions of the index. Log in to one of the node hosts, navigate to {es} installation directory, and follow these steps: - . make sure the repository holding the `.security` snapshot is installed - + +. Make sure the repository holding the `.security` snapshot is installed. ++ -- +For example: [source,shell] -------------------------------------------------- GET /_snapshot/my_backup -------------------------------------------------- -- - . add a new user with the `superuser` role to the file based realm +. Add a new user with the `superuser` role to the file based realm. + -- +For example, create a user named `jacknich`: [source,shell] -------------------------------------------------- bin/elasticsearch-users useradd jacknich -p password -r superuser -------------------------------------------------- -- - . using the previously create user, delete the existing `.security-6` or -`.security-7` index +. Using the previously create user, delete the existing `.security-6` or +`.security-7` index. + -- [source,shell] -------------------------------------------------- - curl -u jacknich-X DELETE "localhost:9200/.security-7" +curl -u jacknich-X DELETE "localhost:9200/.security-7" -------------------------------------------------- -NOTE: after this step authentication will not work so all API calls will fail +NOTE: After this step authentication will not work; all API calls will fail. + -- - . using the same user, restore the `.security` index from the snapshot + . Using the same user, restore the `.security` index from the snapshot. + -- [source,shell] @@ -218,6 +224,7 @@ NOTE: after this step authentication will not work so all API calls will fail -------------------------------------------------- NOTE: Restoring the global state is optional, but it will help make sure the + -- The next part is really common to every other backup restore; copy the backed-up From 002b83a3108bfaca30abbb542b502b973585ddbb Mon Sep 17 00:00:00 2001 From: lcawl Date: Wed, 12 Jun 2019 15:17:22 -0700 Subject: [PATCH 07/42] [DOCS] Edits restore steps --- .../backup-security-configuration.asciidoc | 76 ++++++++++--------- 1 file changed, 41 insertions(+), 35 deletions(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index 65e565ce1bd3f..649ee028974e5 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -147,38 +147,37 @@ NOTE: Transient settings are not considered for backup. [[backup-security-repos]] ===== Controlling access to the backup repository -Simply exposing the contents of the `.security` index, by having a third party -read the snapshot, does not equate to a compromised cluster security. This is -because an attacker still needs to brute force cryptographic hashes before it -can impersonate users or use API keys. But every document in the security index -is sensitive to changes, meaning that any modification has the potential to -compromise the integrity of all the data in the cluster. Therefore it is -recommended to: - -* snapshot `.security` in a dedicated repository, to which read and write -access is strictly restricted and audited -* if there are indications that the snapshot has been read, proceed to changing -the passwords of the users in the native realm as well as revoking API keys -* never restore a snapshot for which there are indications that it has been -tampered with; there is currently no option for the restore process to detect -malicious tampering - -The same recommendations are available for the storing of backups for -configuration files. However, special caution must be employed when storing -the `elasticsearch.keystore` and private key file backups, because currently -the secure store is *not* password protected and any read access can compromise -passwords of external systems and private keys. +Exposing the contents of the `.security` index by having a third party read the +snapshot does not equate to compromised cluster security. An attacker would +still need to brute force cryptographic hashes before it can impersonate users +or use API keys. However, every document in the security index is sensitive to +changes. Modifications to that index have the potential to compromise the +integrity of all the data in the cluster. To prevent that situation: + +* Snapshot the `.security` index in a dedicated repository, where read and write +access is strictly restricted and audited. +* If there are indications that the snapshot has been read, change the passwords +of the users in the native realm and revoke API keys. +* If there are indications that the snapshot has been tampered with, do not +restore it. There is currently no option for the restore process to detect +malicious tampering. + +The same recommendations apply to storing backups for configuration files. +However, use special caution when you store the `elasticsearch.keystore` and +private key file backups. Currently, the secure store is *not* password +protected and any read access can compromise the passwords of external systems +and private keys. [float] [[restore-security-configuration]] ==== Restoring security configuration -The first part of restoring a Security configuration backup is restoring the -`.security` index. You can restore a snapshot of the `.security` index only if -it has been created in the same major release version. The last minor version of -every major release can convert and read both versions of the index. -Log in to one of the node hosts, navigate to {es} installation directory, and -follow these steps: +NOTE: You can restore a snapshot of the `.security` index only if it was created +in the same major release version. The last minor version of every major release +can convert and read both versions of the index. + +To restore your security configuration from a backup, log in to one of the node +hosts, navigate to {es} installation directory, and follow these steps: . Make sure the repository holding the `.security` snapshot is installed. + @@ -189,7 +188,8 @@ For example: GET /_snapshot/my_backup -------------------------------------------------- -- -. Add a new user with the `superuser` role to the file based realm. + +. Add a new user with the `superuser` built-in role to the file based realm. + -- For example, create a user named `jacknich`: @@ -198,7 +198,8 @@ For example, create a user named `jacknich`: bin/elasticsearch-users useradd jacknich -p password -r superuser -------------------------------------------------- -- -. Using the previously create user, delete the existing `.security-6` or + +. Using the previously created user, delete the existing `.security-6` or `.security-7` index. + -- @@ -207,10 +208,11 @@ For example, create a user named `jacknich`: curl -u jacknich-X DELETE "localhost:9200/.security-7" -------------------------------------------------- -NOTE: After this step authentication will not work; all API calls will fail. +WARNING: After this step authentication will not work; all API calls will fail. -- - . Using the same user, restore the `.security` index from the snapshot. + +. Using the same user, restore the `.security` index from the snapshot. + -- [source,shell] @@ -227,8 +229,12 @@ NOTE: Restoring the global state is optional, but it will help make sure the -- -The next part is really common to every other backup restore; copy the backed-up -configuration overwriting the contents of `$ES_PATH_CONF` and restart the node. +. Copy the backup configuration details by overwriting the contents of +`$ES_PATH_CONF` and restart the node. ++ +-- +This step is common to every other backup restore. +-- -Lastly, you can now cherry-pick and <> from earlier. +. Cherry-pick and +<> from earlier. From d27012ac884d1c316f0d82a92468658b938f0153 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Mon, 17 Jun 2019 12:22:40 -0400 Subject: [PATCH 08/42] Update x-pack/docs/en/security/backup-security-configuration.asciidoc Co-Authored-By: Lisa Cawley --- x-pack/docs/en/security/backup-security-configuration.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index 649ee028974e5..4688f8bf55578 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -71,7 +71,7 @@ file, are extra sensitive; they require restricted read access. [float] [[backup-security-snapshots]] -==== Backing up API-based configuration +==== Snapshotting security indices and global cluster metadata To back up API-based configuration, you use the <>. The snapshot API saves the From e394573a5fbb179a3e2bb0c41bc9318e0decba2d Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Mon, 17 Jun 2019 12:23:02 -0400 Subject: [PATCH 09/42] Update x-pack/docs/en/security/backup-security-configuration.asciidoc Co-Authored-By: Lisa Cawley --- x-pack/docs/en/security/backup-security-configuration.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index 4688f8bf55578..c90d11e439fbf 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -147,7 +147,7 @@ NOTE: Transient settings are not considered for backup. [[backup-security-repos]] ===== Controlling access to the backup repository -Exposing the contents of the `.security` index by having a third party read the +If a third party reads the snapshot and exposes the contents of the `.security` index, snapshot does not equate to compromised cluster security. An attacker would still need to brute force cryptographic hashes before it can impersonate users or use API keys. However, every document in the security index is sensitive to From e323ad303e37f6f1966c75afd99ead3f3f125bd7 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Mon, 17 Jun 2019 12:23:15 -0400 Subject: [PATCH 10/42] Update x-pack/docs/en/security/backup-security-configuration.asciidoc Co-Authored-By: Lisa Cawley --- x-pack/docs/en/security/backup-security-configuration.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index c90d11e439fbf..f922987ad593f 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -148,7 +148,7 @@ NOTE: Transient settings are not considered for backup. ===== Controlling access to the backup repository If a third party reads the snapshot and exposes the contents of the `.security` index, -snapshot does not equate to compromised cluster security. An attacker would +it does not mean you have compromised cluster security. An attacker would still need to brute force cryptographic hashes before it can impersonate users or use API keys. However, every document in the security index is sensitive to changes. Modifications to that index have the potential to compromise the From 8b2365267492a59a06038c8a44d8742d617633d3 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Mon, 17 Jun 2019 12:23:25 -0400 Subject: [PATCH 11/42] Update x-pack/docs/en/security/backup-security-configuration.asciidoc Co-Authored-By: Lisa Cawley --- x-pack/docs/en/security/backup-security-configuration.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index f922987ad593f..850782b3e785d 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -149,7 +149,7 @@ NOTE: Transient settings are not considered for backup. If a third party reads the snapshot and exposes the contents of the `.security` index, it does not mean you have compromised cluster security. An attacker would -still need to brute force cryptographic hashes before it can impersonate users +still need to brute force cryptographic hashes before they can impersonate users or use API keys. However, every document in the security index is sensitive to changes. Modifications to that index have the potential to compromise the integrity of all the data in the cluster. To prevent that situation: From 4271692ddcdc9fa0b4baefcb8d267d6bdf5b0429 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Mon, 17 Jun 2019 12:24:34 -0400 Subject: [PATCH 12/42] Update x-pack/docs/en/security/backup-security-configuration.asciidoc Co-Authored-By: Lisa Cawley --- x-pack/docs/en/security/backup-security-configuration.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index 850782b3e785d..8ee5da4d7805b 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -127,7 +127,7 @@ PUT /_snapshot/my_backup/snapshot_1 -------------------------------------------------- <1> A few security settings are stored in the global cluster metadata as -persistent settings. These are included in the snapshot by, but for facilitating +persistent settings. These are included in the snapshot by default, but for facilitating change reviews as well as restore convenience, we can pull them out to stand in their own file. From f1f4cbf17776fee5ab6efb03e4989c507e5d4167 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Mon, 17 Jun 2019 20:47:02 +0300 Subject: [PATCH 13/42] Mooore changes --- .../backup-security-configuration.asciidoc | 87 ++++++++++--------- 1 file changed, 48 insertions(+), 39 deletions(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index 8ee5da4d7805b..c40550c475ae1 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -6,11 +6,12 @@ The {es} {security-features} are configured by files and by various API endpoints. The security settings, which are prefixed with `xpack.security`, are ordinarily included in regular backups. However, the backup is not complete until a special index, which holds most of the non-setting configuration, is -also included. +also included. Hence, the backup and restore of the security configuration +entails an extra step to a conventional backup and restore. [float] -[[backup-security-location]] -==== Location of security configuration details +[[backup-security-files-location]] +==== Location of security configuration files The configuration files reside on every node and are usually identical across nodes. They are formatted as YAML text files. Also, on every node there is a @@ -18,8 +19,13 @@ secure store file named `elasticsearch.keystore`, which is binary formatted. This file is used only for sensitive settings such as service passwords. Some setting values are paths pointing to other files, such as the keys and certificates to configure TLS. Reassuringly, all these files are stored inside -the `$ES_PATH_CONF` directory, so when you back up the whole directory, you -capture all of the file-based configuration for the {security-features}. +the `$ES_PATH_CONF` directory, so when you back up the whole directory, as part +of a conventional backup, you capture all of the file-based configuration for +the {security-features}. + +[float] +[[backup-security-configuration-inside-cluster]] +==== Security configuration inside the cluster The API calls act on configuration information that is stored in the global cluster metadata as well as inside a dedicated {es} index. This index is named @@ -49,20 +55,19 @@ the security details. ==== Backing up configuration files The simplest way to backup configuration files is to copy the `$ES_PATH_CONF` -directory. Overall, this procedure is identical to a complete backup. +directory. Overall, this procedure is identical to a conventional backup. -TIP: In a typical {es} deployment, all the configuration files are identical on +NOTE: In a typical {es} deployment, all the configuration files are identical on each node. It is recommended that you back up the configuration files every time they are -modified. In a typical deployment, after you achieve a desired configuration, -you update these files infrequently. If you change the contents of the -configuration files, the changes usually don't take effect until you restart -the node. Therefore, configuration files should contain only the settings that +modified. This is not as difficult as it sounds because, in a typical +deployment, after you achieve a desired configuration, you update these files +infrequently. Configuration files purposefully contain only the settings that change rarely or are required for cluster formation or require file system privileges. -For the YAML text files, you also have the option of storing diffs against a +TIP: For the YAML text files, you also have the option of storing diffs against a baseline or checking them into `git`. This method makes it easier for the {es} administrator to review the changes. @@ -71,16 +76,17 @@ file, are extra sensitive; they require restricted read access. [float] [[backup-security-snapshots]] -==== Snapshotting security indices and global cluster metadata +==== Snapshotting the security index and the global cluster metadata -To back up API-based configuration, you use the -<>. The snapshot API saves the -settings, the type mapping, and the contents of any index to a location on a -cloud storage service or on a shared network file system. The snapshot operation -can also optionally capture the persistent settings from the global cluster -metadata. +To back up API-based configuration, which is stored inside the cluster, you use +the <>. The snapshot API +saves the settings, the type mapping, and the contents of any index to a +location on a cloud storage service or on a shared network file system. This is +used to snapshot the `.security` index. The snapshot operation can also +optionally capture the persistent settings from the global cluster metadata. . Create a repository that you can use to backup the `.security` index. +Ideally this repository is dedicated only to this scope. + -- You must have the elevated `manage` cluster privilege to create it, yet it's a @@ -127,9 +133,9 @@ PUT /_snapshot/my_backup/snapshot_1 -------------------------------------------------- <1> A few security settings are stored in the global cluster metadata as -persistent settings. These are included in the snapshot by default, but for facilitating -change reviews as well as restore convenience, we can pull them out to stand in -their own file. +persistent settings. These are included in the snapshot by default, but for +facilitating change reviews as well as restore convenience, we can pull them +out to stand in their own file. [source,shell] -------------------------------------------------- @@ -147,12 +153,13 @@ NOTE: Transient settings are not considered for backup. [[backup-security-repos]] ===== Controlling access to the backup repository -If a third party reads the snapshot and exposes the contents of the `.security` index, -it does not mean you have compromised cluster security. An attacker would -still need to brute force cryptographic hashes before they can impersonate users -or use API keys. However, every document in the security index is sensitive to -changes. Modifications to that index have the potential to compromise the -integrity of all the data in the cluster. To prevent that situation: +If a third party reads the snapshot and exposes the contents of the `.security` +index, it does not mean you have compromised cluster security. An attacker +would still need to brute force cryptographic hashes before they can +impersonate users or use API keys. However, every document in the security +index is sensitive to changes. Modifications to that index have the potential +to compromise the integrity of all the data in the cluster. To prevent this +situation: * Snapshot the `.security` index in a dedicated repository, where read and write access is strictly restricted and audited. @@ -205,7 +212,7 @@ For example, create a user named `jacknich`: -- [source,shell] -------------------------------------------------- -curl -u jacknich-X DELETE "localhost:9200/.security-7" +curl -u jacknich -X DELETE "localhost:9200/.security-7" -------------------------------------------------- WARNING: After this step authentication will not work; all API calls will fail. @@ -225,16 +232,18 @@ WARNING: After this step authentication will not work; all API calls will fail. ' -------------------------------------------------- -NOTE: Restoring the global state is optional, but it will help make sure the - +NOTE: The `include_global_state` is optional but it will help to make sure the +configuration in the index is compatible with the rest of the configuration, as +was the case at the time of the snapshot. But be advised that this will also +restore non-security persistent cluster seetting. -- -. Copy the backup configuration details by overwriting the contents of -`$ES_PATH_CONF` and restart the node. -+ --- -This step is common to every other backup restore. --- +. Optionally, cherry-pick and <> that you have extracted with the `GET _cluster/settings` API, if you +need to review and override the settings that were included in the snapshot (by +the `include_global_state` flag). -. Cherry-pick and -<> from earlier. +Lastly, if your backup included configuration files copy these and overwrite +the contents of `$ES_PATH_CONF` and restart the node. This needs to be done on +*every node*. Usually rolling restarts won't work, a full cluster restart is +required. This is a conventional file configuration restore. From 35d64db1504acc702bfc4ae0b48b7c36bcb89c01 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Tue, 18 Jun 2019 00:36:25 +0300 Subject: [PATCH 14/42] CONSOLE --- .../en/security/backup-security-configuration.asciidoc | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index c40550c475ae1..891b1c8828834 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -115,6 +115,7 @@ POST /_security/user/snapshot_user "roles" : [ "snapshot_user" ] } -------------------------------------------------- +// CONSOLE -- . Create a snapshot. @@ -131,6 +132,7 @@ PUT /_snapshot/my_backup/snapshot_1 "include_global_state": true <1> } -------------------------------------------------- +// CONSOLE <1> A few security settings are stored in the global cluster metadata as persistent settings. These are included in the snapshot by default, but for @@ -141,7 +143,6 @@ out to stand in their own file. -------------------------------------------------- curl -X GET -u elastic "localhost:9200/_cluster/settings?pretty&flat_settings" | jq '.persistent | with_entries(select(.key|startswith("xpack.security")))' -------------------------------------------------- -// NOTCONSOLE You can store the output of this together with the other configuration files. @@ -190,10 +191,11 @@ hosts, navigate to {es} installation directory, and follow these steps: + -- For example: -[source,shell] +[source,js] -------------------------------------------------- GET /_snapshot/my_backup -------------------------------------------------- +// CONSOLE -- . Add a new user with the `superuser` built-in role to the file based realm. @@ -202,7 +204,7 @@ GET /_snapshot/my_backup For example, create a user named `jacknich`: [source,shell] -------------------------------------------------- - bin/elasticsearch-users useradd jacknich -p password -r superuser +bin/elasticsearch-users useradd jacknich -p password -r superuser -------------------------------------------------- -- From 3d4730deef969f46dffba458fb1a370c675ff316 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Tue, 18 Jun 2019 12:02:55 +0300 Subject: [PATCH 15/42] Nit --- .../backup-security-configuration.asciidoc | 20 +++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index 891b1c8828834..d0041b5f16264 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -89,6 +89,19 @@ optionally capture the persistent settings from the global cluster metadata. Ideally this repository is dedicated only to this scope. + -- +[source,js] +----------------------------------- +PUT /_snapshot/my_backup +{ + "type": "fs", + "settings": { + "location": "my_backup_location" + } +} +----------------------------------- +// CONSOLE +// TESTSETUP + You must have the elevated `manage` cluster privilege to create it, yet it's a relatively infrequent operation. The elevated privileges are required to prevent regular users from exfiltrating data to locations of their choosing. @@ -190,7 +203,6 @@ hosts, navigate to {es} installation directory, and follow these steps: . Make sure the repository holding the `.security` snapshot is installed. + -- -For example: [source,js] -------------------------------------------------- GET /_snapshot/my_backup @@ -235,9 +247,9 @@ WARNING: After this step authentication will not work; all API calls will fail. -------------------------------------------------- NOTE: The `include_global_state` is optional but it will help to make sure the -configuration in the index is compatible with the rest of the configuration, as -was the case at the time of the snapshot. But be advised that this will also -restore non-security persistent cluster seetting. +configuration in the index is compatible with the rest of the configuration in +the cluster, as was the case at the time of the snapshot. But be advised that +this will also restore non-security persistent cluster seetting. -- . Optionally, cherry-pick and < Date: Mon, 24 Jun 2019 17:55:59 -0400 Subject: [PATCH 16/42] Update x-pack/docs/en/security/backup-security-configuration.asciidoc Co-Authored-By: Tim Vernum --- x-pack/docs/en/security/backup-security-configuration.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index d0041b5f16264..168e8a49c46c3 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -4,7 +4,7 @@ The {es} {security-features} are configured by files and by various API endpoints. The security settings, which are prefixed with `xpack.security`, are -ordinarily included in regular backups. However, the backup is not complete +ordinarily included in regular file-system backups. However, the backup is not complete until a special index, which holds most of the non-setting configuration, is also included. Hence, the backup and restore of the security configuration entails an extra step to a conventional backup and restore. From cce3c9bc5d8bea7cedcf6305cba94c06e57c7234 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Mon, 24 Jun 2019 17:56:30 -0400 Subject: [PATCH 17/42] Update x-pack/docs/en/security/backup-security-configuration.asciidoc Co-Authored-By: Tim Vernum --- x-pack/docs/en/security/backup-security-configuration.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index 168e8a49c46c3..efd441706a9e6 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -7,7 +7,7 @@ endpoints. The security settings, which are prefixed with `xpack.security`, are ordinarily included in regular file-system backups. However, the backup is not complete until a special index, which holds most of the non-setting configuration, is also included. Hence, the backup and restore of the security configuration -entails an extra step to a conventional backup and restore. +requires a combination of file-based backups, and {es} snapshots. [float] [[backup-security-files-location]] From 934c4830757d3a0cbf613cf4507e47465c7f614f Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Mon, 24 Jun 2019 17:59:08 -0400 Subject: [PATCH 18/42] Update x-pack/docs/en/security/backup-security-configuration.asciidoc Co-Authored-By: Tim Vernum --- x-pack/docs/en/security/backup-security-configuration.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index efd441706a9e6..d22014bb0833f 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -86,7 +86,7 @@ used to snapshot the `.security` index. The snapshot operation can also optionally capture the persistent settings from the global cluster metadata. . Create a repository that you can use to backup the `.security` index. -Ideally this repository is dedicated only to this scope. +Ideally this repository is dedicated only to this index. + -- [source,js] From 8ee0b79606e3fffa7b7fe3ab84a9184852eadb1b Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Mon, 24 Jun 2019 17:59:37 -0400 Subject: [PATCH 19/42] Update x-pack/docs/en/security/backup-security-configuration.asciidoc Co-Authored-By: Tim Vernum --- x-pack/docs/en/security/backup-security-configuration.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index d22014bb0833f..82745d10ed1fa 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -80,7 +80,7 @@ file, are extra sensitive; they require restricted read access. To back up API-based configuration, which is stored inside the cluster, you use the <>. The snapshot API -saves the settings, the type mapping, and the contents of any index to a +saves the settings, the type mapping, and the contents of nominated indices to a location on a cloud storage service or on a shared network file system. This is used to snapshot the `.security` index. The snapshot operation can also optionally capture the persistent settings from the global cluster metadata. From 6b67c5f4cae6d690ec7598cbf3d56a1f0233b0d4 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Tue, 25 Jun 2019 06:43:46 -0400 Subject: [PATCH 20/42] Update x-pack/docs/en/security/backup-security-configuration.asciidoc Co-Authored-By: Tim Vernum --- x-pack/docs/en/security/backup-security-configuration.asciidoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index 82745d10ed1fa..964ae10485b64 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -31,7 +31,8 @@ The API calls act on configuration information that is stored in the global cluster metadata as well as inside a dedicated {es} index. This index is named `.security-6` in the 6.x versions and `.security-7` in the 7.x releases. The `.security` alias always points to the appropriate index. This index contains -the data which is *not* part of an ordinary configuration backup. This data +the data which is not available in configuration files and *cannot* be +reliably backed up using standard filesystem tools. This data describes: * the definition of users in the native realm (including hashed passwords) From 91b73028f904418667a8da21166f26fbd505e92d Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Tue, 25 Jun 2019 06:46:34 -0400 Subject: [PATCH 21/42] Update x-pack/docs/en/security/backup-security-configuration.asciidoc Co-Authored-By: Tim Vernum --- .../docs/en/security/backup-security-configuration.asciidoc | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index 964ae10485b64..76f4624b90ea1 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -41,7 +41,11 @@ describes: * application privileges * API keys -Be advised that the index format is only compatible over a single major version. +NOTE: The index format is only compatible within a single major version, +and cannot be restored onto a version earlier that the version from which +it originated. +e.g. You can restore a security snapshot from 6.6.0 into 6.7.0 cluster, but +you cannot restore it to a cluster running {es} 6.5.0 or 7.0.0. TIP: The `elasticsearch.yml` and `elasticsearch.keystore` files and the global cluster metadata contain settings for all the {stack} features. The From 6c574392abbc228273d7c7c3750a00cf4e8b1e46 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Tue, 25 Jun 2019 06:58:40 -0400 Subject: [PATCH 22/42] Update x-pack/docs/en/security/backup-security-configuration.asciidoc Co-Authored-By: Tim Vernum --- x-pack/docs/en/security/backup-security-configuration.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index 76f4624b90ea1..6fe98b05a8d1c 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -84,7 +84,7 @@ file, are extra sensitive; they require restricted read access. ==== Snapshotting the security index and the global cluster metadata To back up API-based configuration, which is stored inside the cluster, you use -the <>. The snapshot API +the <>. The snapshot API saves the settings, the type mapping, and the contents of nominated indices to a location on a cloud storage service or on a shared network file system. This is used to snapshot the `.security` index. The snapshot operation can also From ad7ef35a60e505d9f96e905f2e5b9b3e37c2c9fd Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Tue, 25 Jun 2019 07:00:05 -0400 Subject: [PATCH 23/42] Update x-pack/docs/en/security/backup-security-configuration.asciidoc Co-Authored-By: Tim Vernum --- x-pack/docs/en/security/backup-security-configuration.asciidoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index 6fe98b05a8d1c..7c2d89bc63d8b 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -86,7 +86,8 @@ file, are extra sensitive; they require restricted read access. To back up API-based configuration, which is stored inside the cluster, you use the <>. The snapshot API saves the settings, the type mapping, and the contents of nominated indices to a -location on a cloud storage service or on a shared network file system. This is +location on a cloud storage service or on a shared network file system. +This API should be used to snapshot the `.security` index. The snapshot operation can also optionally capture the persistent settings from the global cluster metadata. From e6bb10552d84fad2317460602b3853b1cdb65b1a Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Tue, 25 Jun 2019 07:06:42 -0400 Subject: [PATCH 24/42] Update x-pack/docs/en/security/backup-security-configuration.asciidoc Co-Authored-By: Tim Vernum --- .../en/security/backup-security-configuration.asciidoc | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index 7c2d89bc63d8b..8238e15be0c5d 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -265,5 +265,12 @@ the `include_global_state` flag). Lastly, if your backup included configuration files copy these and overwrite the contents of `$ES_PATH_CONF` and restart the node. This needs to be done on -*every node*. Usually rolling restarts won't work, a full cluster restart is +*every node*. Depending on the extent of the differences between your current +cluster configuration and the restored configuration, you may not be able to +perform a rolling restart. +If you are performing a full restore of your configuration directory, we recommend +a full cluster restart as the safest option. +Alternatively, you may wish to restore you configuration files to a separate +location on disk and use file comparison tools to review the differences between +your existing configuration and the restored configuration. required. This is a conventional file configuration restore. From ad4136e5339c5130b3301aaa611a950437a18fea Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Tue, 25 Jun 2019 07:08:28 -0400 Subject: [PATCH 25/42] Update x-pack/docs/en/security/backup-security-configuration.asciidoc Co-Authored-By: Tim Vernum --- x-pack/docs/en/security/backup-security-configuration.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index 8238e15be0c5d..68e688301fd6f 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -255,7 +255,7 @@ WARNING: After this step authentication will not work; all API calls will fail. NOTE: The `include_global_state` is optional but it will help to make sure the configuration in the index is compatible with the rest of the configuration in the cluster, as was the case at the time of the snapshot. But be advised that -this will also restore non-security persistent cluster seetting. +this will also restore non-security persistent cluster settings. -- . Optionally, cherry-pick and < Date: Tue, 25 Jun 2019 07:25:42 -0400 Subject: [PATCH 26/42] Update x-pack/docs/en/security/backup-security-configuration.asciidoc Co-Authored-By: Tim Vernum --- x-pack/docs/en/security/backup-security-configuration.asciidoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index 68e688301fd6f..156fa95d68b91 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -140,7 +140,8 @@ POST /_security/user/snapshot_user . Create a snapshot. + -- -The following example exemplifies the create snapshot operation, inside the +The following example shows how to use the create snapshot API to backup +the security index to the `my_backup` repository. `my_backup` repository: [source,js] From 86635827a6477a15da1df9a1defa75053ee68d72 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Tue, 25 Jun 2019 07:39:51 -0400 Subject: [PATCH 27/42] Update x-pack/docs/en/security/backup-security-configuration.asciidoc Co-Authored-By: Tim Vernum --- .../en/security/backup-security-configuration.asciidoc | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc index 156fa95d68b91..215dbebfd614d 100644 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ b/x-pack/docs/en/security/backup-security-configuration.asciidoc @@ -236,7 +236,12 @@ bin/elasticsearch-users useradd jacknich -p password -r superuser curl -u jacknich -X DELETE "localhost:9200/.security-7" -------------------------------------------------- -WARNING: After this step authentication will not work; all API calls will fail. +WARNING: After this step any authentication that relies on the `.security` +index will not work. This means that all API calls that authenticate with +native or reserved users will fail, as will any user that relies on a native role. +The file-realm user we created in the step above will continue to work +because it is not stored in the security index, and uses the builtin +`superuser` role. -- From 7d43bf82fae665a1cd84936b8ecc5baf2ce47f53 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Wed, 26 Jun 2019 11:47:35 +0300 Subject: [PATCH 28/42] Generalize data backup and configuration backup --- docs/reference/administering.asciidoc | 72 +++++++++++++++++++++++++-- 1 file changed, 69 insertions(+), 3 deletions(-) diff --git a/docs/reference/administering.asciidoc b/docs/reference/administering.asciidoc index 7170185c6ef77..ecf539aee308f 100644 --- a/docs/reference/administering.asciidoc +++ b/docs/reference/administering.asciidoc @@ -9,8 +9,8 @@ cluster. -- -[[backup-cluster]] -== Back up a cluster +[[backup-cluster-data]] +== Back up a cluster's data As with any software that stores data, it is important to routinely back up your data. {es} replicas provide high availability during runtime; they enable you to @@ -20,6 +20,72 @@ Replicas do not provide protection from catastrophic failure, however. For that, you need a real backup of your cluster—a complete copy in case something goes wrong. -To back up your cluster, you can use the <>. +To back up your cluster's data, you can use the <>. include::{es-repo-dir}/modules/snapshots.asciidoc[tag=snapshot-intro] + +[role="xpack"] +[[backup-secured-cluster-data]] +== Data backup of a secured cluster + +In this context, a secured cluster designates a cluster with {es} +{security-features} enabled. Specifically, this means that the snapshot API has +to be authorized. + +`snapshot_user` is a reserved role that can be assigned to the user that the +snapshot endpoint is run as. This is the only role necessary if all the user +does is periodic snapshots as part of the backup procedure. Besides, it also +grants privileges to list all the existing snapshots (of any repository) as +well as list and view settings of all indices. It does *not* grant privileges +to create repositories, restore snapshots, or search indices. Hence, the user +can view and snapshot all indices, but cannot access or modify any data. + +The restore API requires the `manage` cluster privilege. There is no bespoke +role for the restore process. This privilege is very permissive and should only +be granted to users in the "administrator" category. Automated tools should not +run as users with this privilege. + +[[backup-cluster-configuration]] +== Back up a cluster's configuration + +Besides data backup, configuration backup is also important, especially when it +becomes large and difficult to reconstruct. + +Configuration resides in <> on every +cluster node. Sensitive setting values, like for example passwords for +Watcher's notification servers, are specified inside a binary secure container, +the <> file. Moreover, some setting +values are file paths to the associated configuration data, for example the +ingest geo ip database. All these files are contained inside the `ES_PATH_CONF` +directory. + +NOTE: All changes to configuration files are done by manually editing the files +or using command line utilities, but *not* through APIs. In practice, these +changes are infrequent after the initial setup. + +We recommend that you take regular (ideally, daily) backups of your {es} config +(`$ES_PATH_CONF`) directory using the file backup software of your choice. +Because some of these files may contain sensitive data such as passwords and +TLS keys, you should investigate whether your backup software and/or storage +solution are able to encrypt this data. + +Using the <> it is possible to +override some settings from configuration files. You can capture these in a +*data* backup snapshot by specifing the `include_global_state: true` (default) +parameter to the snapshot API. Using the <> it is also possible to extract these configuration values in a text +format. + +TIP: We recommend that you have a configuration management plan for these +configuration files. You may wish to check them into version control, or +provision them though your choice of configuration management tool. + +NOTE: {es} {security-features} store configuration data, like for example role +definitions and API keys, inside a dedicate special index. This "system" data, +complements the <> configuration and should +backed up as well. See . + +NOTE: Other {stack} components, like Kibana and {ml-cap}, store their configuration +data inside other dedicated indices. From {es}'s perspective these are just data +so you can use the regular <> process. + From da9c103eaf93137133530455a2c8aacb3eee426d Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Wed, 26 Jun 2019 12:04:27 +0300 Subject: [PATCH 29/42] Administering nits --- docs/reference/administering.asciidoc | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/reference/administering.asciidoc b/docs/reference/administering.asciidoc index ecf539aee308f..1bbc437bc1839 100644 --- a/docs/reference/administering.asciidoc +++ b/docs/reference/administering.asciidoc @@ -29,8 +29,8 @@ include::{es-repo-dir}/modules/snapshots.asciidoc[tag=snapshot-intro] == Data backup of a secured cluster In this context, a secured cluster designates a cluster with {es} -{security-features} enabled. Specifically, this means that the snapshot API has -to be authorized. +{security-features} enabled. Specifically, this means that the snapshot API +call needs to be authorized. `snapshot_user` is a reserved role that can be assigned to the user that the snapshot endpoint is run as. This is the only role necessary if all the user @@ -42,8 +42,9 @@ can view and snapshot all indices, but cannot access or modify any data. The restore API requires the `manage` cluster privilege. There is no bespoke role for the restore process. This privilege is very permissive and should only -be granted to users in the "administrator" category. Automated tools should not -run as users with this privilege. +be granted to users in the "administrator" category. Specifically, it allows +users to exfiltrate data to a location of their choosing. Automated tools +should not run as users with this privilege. [[backup-cluster-configuration]] == Back up a cluster's configuration From 7a5c117215f1a1224200e39af89c95c0847409da Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Wed, 26 Jun 2019 16:41:39 +0300 Subject: [PATCH 30/42] RM backup-security-configuration.asciidoc --- docs/reference/administering.asciidoc | 265 +++++++++++++++- .../backup-security-configuration.asciidoc | 282 ------------------ .../docs/en/security/configuring-es.asciidoc | 1 - 3 files changed, 263 insertions(+), 285 deletions(-) delete mode 100644 x-pack/docs/en/security/backup-security-configuration.asciidoc diff --git a/docs/reference/administering.asciidoc b/docs/reference/administering.asciidoc index 1bbc437bc1839..bdc333fd1af4e 100644 --- a/docs/reference/administering.asciidoc +++ b/docs/reference/administering.asciidoc @@ -46,6 +46,9 @@ be granted to users in the "administrator" category. Specifically, it allows users to exfiltrate data to a location of their choosing. Automated tools should not run as users with this privilege. +For more information, see {stack-ov}/security-privileges.html[Security privileges] +and {stack-ov}/built-in-roles.html[Built-in roles]. + [[backup-cluster-configuration]] == Back up a cluster's configuration @@ -75,7 +78,19 @@ override some settings from configuration files. You can capture these in a *data* backup snapshot by specifing the `include_global_state: true` (default) parameter to the snapshot API. Using the <> it is also possible to extract these configuration values in a text -format. +format: + +[source,js] +-------------------------------------------------- +GET _cluster/settings?pretty&flat_settings&filter_path=persistent +-------------------------------------------------- +//CONSOLE +//TEST + +You can store the output of this as a file together with the rest of +configuration files. + +NOTE: Transient settings are not considered for backup. TIP: We recommend that you have a configuration management plan for these configuration files. You may wish to check them into version control, or @@ -84,9 +99,255 @@ provision them though your choice of configuration management tool. NOTE: {es} {security-features} store configuration data, like for example role definitions and API keys, inside a dedicate special index. This "system" data, complements the <> configuration and should -backed up as well. See . +<>. NOTE: Other {stack} components, like Kibana and {ml-cap}, store their configuration data inside other dedicated indices. From {es}'s perspective these are just data so you can use the regular <> process. +[role="xpack"] +[[security-backup]] +== Backup security configuration + +[role="xpack"] +[[backup-security-file-based-configuration]] +=== Backup file-based security configuration + +{es} {security-features} are configured using the <> inside the `elasticsearch.yml` and +`elasticsearch.keystore` files. In addition there are several other +<> inside the same `ES_PATH_CONF` +directory. These files define roles and role mappings as well as +<>. More, some of the +settings specify file paths to security sensitive data, such as TLS keys and +certificates for the HTTP client and inter-node communication and private key files for +<>, <> and the +<> realms. All these are also stored inside +`ES_PATH_CONF` (the path settings are relative). + +To back up all this configuration you can use a <>. + +NOTE: File backups must run on every cluster node. + +NOTE: File backups will store non-security configuration as well. Backing-up +only {security-features} configurations is not supported. A backup is a +point in time record of state of the complete configuration. + +IMPORTANT: The `elasticsearch.keystore`, TLS keys and SAML, OIDC, and Kerberos +realms private key files require confidentiality. This is crucial when files +are copied to the backup location, as this increases the surface for malicious +snooping. + +[float] +[[backup-security-index-configuration]] +=== Backup security configuration data + +{es} {security-features} store configuration or "system" data inside a +dedicated index. This index is named `.security-6` in the {es} 6.x versions and +`.security-7` in the 7.x releases. The `.security` alias always points to the +appropriate index. This index contains the data which is not available in +configuration files and *cannot* be reliably backed up using standard +filesystem tools. This data describes: + +* the definition of users in the native realm (including hashed passwords) +* role definitions (defined via the <>) +* role mappings (defined via the <>) +* application privileges +* API keys + +Hence, `.security` contains resources and definitions more than configurations, +but all of which are required in a complete {security-features} backup. + +Use the <> to backup +`.security`, as you would for any <>. + +NOTE: The index format is only compatible within a single major version, +and cannot be restored onto a version earlier that the version from which +it originated. +e.g. You can restore a security snapshot from 6.6.0 into 6.7.0 cluster, but +you cannot restore it to a cluster running {es} 6.5.0 or 7.0.0. + +[float] +==== Snapshot .security steps + +. Create a repository that you can use to backup the `.security` index. +Ideally this repository is dedicated only to this index. ++ +-- +[source,js] +----------------------------------- +PUT /_snapshot/my_backup +{ + "type": "fs", + "settings": { + "location": "my_backup_location" + } +} +----------------------------------- +// CONSOLE +// TESTSETUP + +You must have the elevated `manage` cluster privilege to create it, yet it's a +relatively infrequent operation. The elevated privileges are required to prevent +regular users from exfiltrating data to locations of their choosing. + +Afterwards, users with the limited `snapshot_user` built-in role can create +snapshots inside that repository. This role grants privileges to create +snapshots in any existing repositories. It also grants privileges to list and +read settings for all indices (but not to read data). + +-- + +. Create users that have the `snapshot_user` built-in role. ++ +-- +The following example creates a new user `snapshot_user` in the native realm: + +[source,js] +-------------------------------------------------- +POST /_security/user/snapshot_user +{ + "password" : "secret", + "roles" : [ "snapshot_user" ] +} +-------------------------------------------------- +// CONSOLE +-- + +. Create a snapshot. ++ +-- +The following example shows how to use the create snapshot API to backup +the security index to the `my_backup` repository. +`my_backup` repository: + +[source,js] +-------------------------------------------------- +PUT /_snapshot/my_backup/snapshot_1 +{ + "indices": ".security", + "include_global_state": true <1> +} +-------------------------------------------------- +// CONSOLE + +<1> A few security settings are stored in the global cluster metadata as +persistent settings. These are included in the snapshot by default, but for +facilitating change reviews as well as restore convenience, we can pull them +out to stand in their own file. + +used to snapshot the `.security` index. The snapshot operation can also +optionally capture the persistent settings from the global cluster metadata. +-- + +[float] +[[backup-security-repos]] +===== Controlling access to the backup repository + +If a third party reads the snapshot and exposes the contents of the `.security` +index, it does not mean you have compromised cluster security. An attacker +would still need to brute force cryptographic hashes before they can +impersonate users or use API keys. However, every document in the security +index is sensitive to changes. Modifications to that index have the potential +to compromise the integrity of all the data in the cluster. To prevent this +situation: + +* Snapshot the `.security` index in a dedicated repository, where read and write +access is strictly restricted and audited. +* If there are indications that the snapshot has been read, change the passwords +of the users in the native realm and revoke API keys. +* If there are indications that the snapshot has been tampered with, do not +restore it. There is currently no option for the restore process to detect +malicious tampering. + +The same recommendations apply to storing backups for configuration files. +However, use special caution when you store the `elasticsearch.keystore` and +private key file backups. Currently, the secure store is *not* password +protected and any read access can compromise the passwords of external systems +and private keys. + +[float] +[[restore-security-configuration]] +==== Restoring security configuration + +NOTE: You can restore a snapshot of the `.security` index only if it was created +in the same major release version. The last minor version of every major release +can convert and read both versions of the index. + +To restore your security configuration from a backup, log in to one of the node +hosts, navigate to {es} installation directory, and follow these steps: + +. Make sure the repository holding the `.security` snapshot is installed. ++ +-- +[source,js] +-------------------------------------------------- +GET /_snapshot/my_backup +-------------------------------------------------- +// CONSOLE +-- + +. Add a new user with the `superuser` built-in role to the file based realm. ++ +-- +For example, create a user named `jacknich`: +[source,shell] +-------------------------------------------------- +bin/elasticsearch-users useradd jacknich -p password -r superuser +-------------------------------------------------- +-- + +. Using the previously created user, delete the existing `.security-6` or +`.security-7` index. ++ +-- +[source,shell] +-------------------------------------------------- +curl -u jacknich -X DELETE "localhost:9200/.security-7" +-------------------------------------------------- + +WARNING: After this step any authentication that relies on the `.security` +index will not work. This means that all API calls that authenticate with +native or reserved users will fail, as will any user that relies on a native role. +The file-realm user we created in the step above will continue to work +because it is not stored in the security index, and uses the builtin +`superuser` role. + +-- + +. Using the same user, restore the `.security` index from the snapshot. ++ +-- +[source,shell] +-------------------------------------------------- + curl -u jacknich -X POST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -H 'Content-Type: application/json' -d' + { + "indices": ".security-7", + "include_global_state": true + } + ' +-------------------------------------------------- + +NOTE: The `include_global_state` is optional but it will help to make sure the +configuration in the index is compatible with the rest of the configuration in +the cluster, as was the case at the time of the snapshot. But be advised that +this will also restore non-security persistent cluster settings. +-- + +. Optionally, cherry-pick and <> that you have extracted with the `GET _cluster/settings` API, if you +need to review and override the settings that were included in the snapshot (by +the `include_global_state` flag). + +Lastly, if your backup included configuration files copy these and overwrite +the contents of `$ES_PATH_CONF` and restart the node. This needs to be done on +*every node*. Depending on the extent of the differences between your current +cluster configuration and the restored configuration, you may not be able to +perform a rolling restart. +If you are performing a full restore of your configuration directory, we recommend +a full cluster restart as the safest option. +Alternatively, you may wish to restore you configuration files to a separate +location on disk and use file comparison tools to review the differences between +your existing configuration and the restored configuration. +required. This is a conventional file configuration restore. diff --git a/x-pack/docs/en/security/backup-security-configuration.asciidoc b/x-pack/docs/en/security/backup-security-configuration.asciidoc deleted file mode 100644 index 215dbebfd614d..0000000000000 --- a/x-pack/docs/en/security/backup-security-configuration.asciidoc +++ /dev/null @@ -1,282 +0,0 @@ -[role="xpack"] -[[backup-restore-security-configuration]] -=== Backing up and restoring security configuration - -The {es} {security-features} are configured by files and by various API -endpoints. The security settings, which are prefixed with `xpack.security`, are -ordinarily included in regular file-system backups. However, the backup is not complete -until a special index, which holds most of the non-setting configuration, is -also included. Hence, the backup and restore of the security configuration -requires a combination of file-based backups, and {es} snapshots. - -[float] -[[backup-security-files-location]] -==== Location of security configuration files - -The configuration files reside on every node and are usually identical across -nodes. They are formatted as YAML text files. Also, on every node there is a -secure store file named `elasticsearch.keystore`, which is binary formatted. -This file is used only for sensitive settings such as service passwords. Some -setting values are paths pointing to other files, such as the keys and -certificates to configure TLS. Reassuringly, all these files are stored inside -the `$ES_PATH_CONF` directory, so when you back up the whole directory, as part -of a conventional backup, you capture all of the file-based configuration for -the {security-features}. - -[float] -[[backup-security-configuration-inside-cluster]] -==== Security configuration inside the cluster - -The API calls act on configuration information that is stored in the global -cluster metadata as well as inside a dedicated {es} index. This index is named -`.security-6` in the 6.x versions and `.security-7` in the 7.x releases. The -`.security` alias always points to the appropriate index. This index contains -the data which is not available in configuration files and *cannot* be -reliably backed up using standard filesystem tools. This data -describes: - -* the definition of users in the native realm (including hashed passwords) -* role definitions -* role mappings -* application privileges -* API keys - -NOTE: The index format is only compatible within a single major version, -and cannot be restored onto a version earlier that the version from which -it originated. -e.g. You can restore a security snapshot from 6.6.0 into 6.7.0 cluster, but -you cannot restore it to a cluster running {es} 6.5.0 or 7.0.0. - -TIP: The `elasticsearch.yml` and `elasticsearch.keystore` files and the global -cluster metadata contain settings for all the {stack} features. The -`.security` index and all the other -<> contain configuration data -exclusively for the {security-features}. It is easiest to backup and restore all -the configuration information as a whole, rather than trying to cherry-pick only -the security details. - -[float] -[[backup-security-files]] -==== Backing up configuration files - -The simplest way to backup configuration files is to copy the `$ES_PATH_CONF` -directory. Overall, this procedure is identical to a conventional backup. - -NOTE: In a typical {es} deployment, all the configuration files are identical on -each node. - -It is recommended that you back up the configuration files every time they are -modified. This is not as difficult as it sounds because, in a typical -deployment, after you achieve a desired configuration, you update these files -infrequently. Configuration files purposefully contain only the settings that -change rarely or are required for cluster formation or require file system -privileges. - -TIP: For the YAML text files, you also have the option of storing diffs against a -baseline or checking them into `git`. This method makes it easier for the {es} -administrator to review the changes. - -The binary and encoded files, such as the private keys and the secure store -file, are extra sensitive; they require restricted read access. - -[float] -[[backup-security-snapshots]] -==== Snapshotting the security index and the global cluster metadata - -To back up API-based configuration, which is stored inside the cluster, you use -the <>. The snapshot API -saves the settings, the type mapping, and the contents of nominated indices to a -location on a cloud storage service or on a shared network file system. -This API should be -used to snapshot the `.security` index. The snapshot operation can also -optionally capture the persistent settings from the global cluster metadata. - -. Create a repository that you can use to backup the `.security` index. -Ideally this repository is dedicated only to this index. -+ --- -[source,js] ------------------------------------ -PUT /_snapshot/my_backup -{ - "type": "fs", - "settings": { - "location": "my_backup_location" - } -} ------------------------------------ -// CONSOLE -// TESTSETUP - -You must have the elevated `manage` cluster privilege to create it, yet it's a -relatively infrequent operation. The elevated privileges are required to prevent -regular users from exfiltrating data to locations of their choosing. - -Afterwards, users with the limited `snapshot_user` built-in role can create -snapshots inside that repository. This role grants privileges to create -snapshots in any existing repositories. It also grants privileges to list and -read settings for all indices (but not to read data). - -For more information, see {stack-ov}/security-privileges.html[Security privileges] -and {stack-ov}/built-in-roles.html[Built-in roles]. --- - -. Create users that have the `snapshot_user` built-in role. -+ --- -The following example creates a new user `snapshot_user` in the native realm: - -[source,js] --------------------------------------------------- -POST /_security/user/snapshot_user -{ - "password" : "secret", - "roles" : [ "snapshot_user" ] -} --------------------------------------------------- -// CONSOLE --- - -. Create a snapshot. -+ --- -The following example shows how to use the create snapshot API to backup -the security index to the `my_backup` repository. -`my_backup` repository: - -[source,js] --------------------------------------------------- -PUT /_snapshot/my_backup/snapshot_1 -{ - "indices": ".security", - "include_global_state": true <1> -} --------------------------------------------------- -// CONSOLE - -<1> A few security settings are stored in the global cluster metadata as -persistent settings. These are included in the snapshot by default, but for -facilitating change reviews as well as restore convenience, we can pull them -out to stand in their own file. - -[source,shell] --------------------------------------------------- -curl -X GET -u elastic "localhost:9200/_cluster/settings?pretty&flat_settings" | jq '.persistent | with_entries(select(.key|startswith("xpack.security")))' --------------------------------------------------- - -You can store the output of this together with the other configuration files. - -NOTE: Transient settings are not considered for backup. - --- - -[float] -[[backup-security-repos]] -===== Controlling access to the backup repository - -If a third party reads the snapshot and exposes the contents of the `.security` -index, it does not mean you have compromised cluster security. An attacker -would still need to brute force cryptographic hashes before they can -impersonate users or use API keys. However, every document in the security -index is sensitive to changes. Modifications to that index have the potential -to compromise the integrity of all the data in the cluster. To prevent this -situation: - -* Snapshot the `.security` index in a dedicated repository, where read and write -access is strictly restricted and audited. -* If there are indications that the snapshot has been read, change the passwords -of the users in the native realm and revoke API keys. -* If there are indications that the snapshot has been tampered with, do not -restore it. There is currently no option for the restore process to detect -malicious tampering. - -The same recommendations apply to storing backups for configuration files. -However, use special caution when you store the `elasticsearch.keystore` and -private key file backups. Currently, the secure store is *not* password -protected and any read access can compromise the passwords of external systems -and private keys. - -[float] -[[restore-security-configuration]] -==== Restoring security configuration - -NOTE: You can restore a snapshot of the `.security` index only if it was created -in the same major release version. The last minor version of every major release -can convert and read both versions of the index. - -To restore your security configuration from a backup, log in to one of the node -hosts, navigate to {es} installation directory, and follow these steps: - -. Make sure the repository holding the `.security` snapshot is installed. -+ --- -[source,js] --------------------------------------------------- -GET /_snapshot/my_backup --------------------------------------------------- -// CONSOLE --- - -. Add a new user with the `superuser` built-in role to the file based realm. -+ --- -For example, create a user named `jacknich`: -[source,shell] --------------------------------------------------- -bin/elasticsearch-users useradd jacknich -p password -r superuser --------------------------------------------------- --- - -. Using the previously created user, delete the existing `.security-6` or -`.security-7` index. -+ --- -[source,shell] --------------------------------------------------- -curl -u jacknich -X DELETE "localhost:9200/.security-7" --------------------------------------------------- - -WARNING: After this step any authentication that relies on the `.security` -index will not work. This means that all API calls that authenticate with -native or reserved users will fail, as will any user that relies on a native role. -The file-realm user we created in the step above will continue to work -because it is not stored in the security index, and uses the builtin -`superuser` role. - --- - -. Using the same user, restore the `.security` index from the snapshot. -+ --- -[source,shell] --------------------------------------------------- - curl -u jacknich -X POST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -H 'Content-Type: application/json' -d' - { - "indices": ".security-7", - "include_global_state": true - } - ' --------------------------------------------------- - -NOTE: The `include_global_state` is optional but it will help to make sure the -configuration in the index is compatible with the rest of the configuration in -the cluster, as was the case at the time of the snapshot. But be advised that -this will also restore non-security persistent cluster settings. --- - -. Optionally, cherry-pick and <> that you have extracted with the `GET _cluster/settings` API, if you -need to review and override the settings that were included in the snapshot (by -the `include_global_state` flag). - -Lastly, if your backup included configuration files copy these and overwrite -the contents of `$ES_PATH_CONF` and restart the node. This needs to be done on -*every node*. Depending on the extent of the differences between your current -cluster configuration and the restored configuration, you may not be able to -perform a rolling restart. -If you are performing a full restore of your configuration directory, we recommend -a full cluster restart as the safest option. -Alternatively, you may wish to restore you configuration files to a separate -location on disk and use file comparison tools to review the differences between -your existing configuration and the restored configuration. -required. This is a conventional file configuration restore. diff --git a/x-pack/docs/en/security/configuring-es.asciidoc b/x-pack/docs/en/security/configuring-es.asciidoc index 55f0a6fe164e3..8fefe2b830ca9 100644 --- a/x-pack/docs/en/security/configuring-es.asciidoc +++ b/x-pack/docs/en/security/configuring-es.asciidoc @@ -163,7 +163,6 @@ include::authentication/configuring-saml-realm.asciidoc[] include::authentication/configuring-kerberos-realm.asciidoc[] :edit_url: -include::backup-security-configuration.asciidoc[] include::{es-repo-dir}/security/reference/files.asciidoc[] include::fips-140-compliance.asciidoc[] From 7e944da2dac6ca73e3443d3a2606061ffdfeb856 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Wed, 26 Jun 2019 19:32:38 +0300 Subject: [PATCH 31/42] Review --- docs/reference/administering.asciidoc | 195 ++++++++++++++------------ 1 file changed, 107 insertions(+), 88 deletions(-) diff --git a/docs/reference/administering.asciidoc b/docs/reference/administering.asciidoc index bdc333fd1af4e..2040da111f282 100644 --- a/docs/reference/administering.asciidoc +++ b/docs/reference/administering.asciidoc @@ -36,15 +36,16 @@ call needs to be authorized. snapshot endpoint is run as. This is the only role necessary if all the user does is periodic snapshots as part of the backup procedure. Besides, it also grants privileges to list all the existing snapshots (of any repository) as -well as list and view settings of all indices. It does *not* grant privileges -to create repositories, restore snapshots, or search indices. Hence, the user -can view and snapshot all indices, but cannot access or modify any data. +well as list and view settings of all indices, including the `.security` index. +It does *not* grant privileges to create repositories, restore snapshots, or +search indices. Hence, the user can view and snapshot all indices, but cannot +access or modify any data. The restore API requires the `manage` cluster privilege. There is no bespoke role for the restore process. This privilege is very permissive and should only be granted to users in the "administrator" category. Specifically, it allows -users to exfiltrate data to a location of their choosing. Automated tools -should not run as users with this privilege. +malicious users to exfiltrate data to a location of their choosing. Automated +tools should not run as users with this privilege. For more information, see {stack-ov}/security-privileges.html[Security privileges] and {stack-ov}/built-in-roles.html[Built-in roles]. @@ -110,6 +111,7 @@ so you can use the regular <> process. == Backup security configuration [role="xpack"] +[float] [[backup-security-file-based-configuration]] === Backup file-based security configuration @@ -139,11 +141,12 @@ realms private key files require confidentiality. This is crucial when files are copied to the backup location, as this increases the surface for malicious snooping. +[role="xpack"] [float] [[backup-security-index-configuration]] -=== Backup security configuration data +=== Backup security data -{es} {security-features} store configuration or "system" data inside a +{es} {security-features} store system configuration data inside a dedicated index. This index is named `.security-6` in the {es} 6.x versions and `.security-7` in the 7.x releases. The `.security` alias always points to the appropriate index. This index contains the data which is not available in @@ -161,18 +164,12 @@ but all of which are required in a complete {security-features} backup. Use the <> to backup `.security`, as you would for any <>. - -NOTE: The index format is only compatible within a single major version, -and cannot be restored onto a version earlier that the version from which -it originated. -e.g. You can restore a security snapshot from 6.6.0 into 6.7.0 cluster, but -you cannot restore it to a cluster running {es} 6.5.0 or 7.0.0. - -[float] -==== Snapshot .security steps +For convenience, here are the complete steps: . Create a repository that you can use to backup the `.security` index. -Ideally this repository is dedicated only to this index. +It is preferable to have a <> for +this special index. + + -- [source,js] @@ -186,23 +183,18 @@ PUT /_snapshot/my_backup } ----------------------------------- // CONSOLE -// TESTSETUP -You must have the elevated `manage` cluster privilege to create it, yet it's a -relatively infrequent operation. The elevated privileges are required to prevent -regular users from exfiltrating data to locations of their choosing. - -Afterwards, users with the limited `snapshot_user` built-in role can create -snapshots inside that repository. This role grants privileges to create -snapshots in any existing repositories. It also grants privileges to list and -read settings for all indices (but not to read data). +The user calling this API must have the elevated `manage` cluster privilege, to +prevent <>. -- -. Create users that have the `snapshot_user` built-in role. +. Create a user and assign it only the built-in `snapshot_user` role. + -- -The following example creates a new user `snapshot_user` in the native realm: +The following example creates a new user `snapshot_user` in the +{stack-ov}/native-realm.html[native realm], but it is not important of which +realm the user is a member of: [source,js] -------------------------------------------------- @@ -213,14 +205,15 @@ POST /_security/user/snapshot_user } -------------------------------------------------- // CONSOLE +// TEST[continued] + -- -. Create a snapshot. +. Create incremental snapshots authorizing as `snapshot_user`. + -- The following example shows how to use the create snapshot API to backup -the security index to the `my_backup` repository. -`my_backup` repository: +the `.security` index to the `my_backup` repository created earlier. [source,js] -------------------------------------------------- @@ -231,27 +224,37 @@ PUT /_snapshot/my_backup/snapshot_1 } -------------------------------------------------- // CONSOLE +// TEST[continued] -<1> A few security settings are stored in the global cluster metadata as -persistent settings. These are included in the snapshot by default, but for -facilitating change reviews as well as restore convenience, we can pull them -out to stand in their own file. +<1> This captures all the persistent settings stored in the global cluster +metadata as well as other configurations such as aliases and stored scripts. +Note that this includes non-security configuration and that it complements but +does not replace the <>. -used to snapshot the `.security` index. The snapshot operation can also -optionally capture the persistent settings from the global cluster metadata. -- +IMPORTANT: The index format is only compatible within a single major version, +and cannot be restored onto a version earlier that the version from which it +originated. e.g. You can restore a security snapshot from 6.6.0 into 6.7.0 +cluster, but you cannot restore it to a cluster running {es} 6.5.0 or 7.0.0. + [float] [[backup-security-repos]] -===== Controlling access to the backup repository +==== Controlling access to the backup repository -If a third party reads the snapshot and exposes the contents of the `.security` -index, it does not mean you have compromised cluster security. An attacker -would still need to brute force cryptographic hashes before they can -impersonate users or use API keys. However, every document in the security -index is sensitive to changes. Modifications to that index have the potential -to compromise the integrity of all the data in the cluster. To prevent this -situation: +The snapshot of the security index will typically contain sensitive data such +as user names and password hashes. Because password are stored using +<> the disclosure of a snapshot would +not automaticaly allow a third party to authenticate as one of your users or +use API keys, but it would disclose confidential information. + +It is also important that you protected the integrity of these backups in case +you ever need to restore them. If a third party is able to modify the stored +backups they may be able to install a back door that would grant access if the +snapshot is loaded into an {es} cluster. + +We recommend that you: * Snapshot the `.security` index in a dedicated repository, where read and write access is strictly restricted and audited. @@ -261,40 +264,52 @@ of the users in the native realm and revoke API keys. restore it. There is currently no option for the restore process to detect malicious tampering. -The same recommendations apply to storing backups for configuration files. -However, use special caution when you store the `elasticsearch.keystore` and -private key file backups. Currently, the secure store is *not* password -protected and any read access can compromise the passwords of external systems -and private keys. - [float] +[role="xpack"] [[restore-security-configuration]] -==== Restoring security configuration +=== Restoring security configuration -NOTE: You can restore a snapshot of the `.security` index only if it was created -in the same major release version. The last minor version of every major release -can convert and read both versions of the index. +NOTE: You can restore a snapshot of the `.security` index only if it was +created in a previous minor version in the same major version. The last minor +version of every major release can convert and read formats of the index for +both its major version and the next one. -To restore your security configuration from a backup, log in to one of the node -hosts, navigate to {es} installation directory, and follow these steps: +When you restore security configuration you have the option of doing a complete +restore of *all* configurations, including non-security ones, or to only restore +the contents of `.security`, which as <> described, comprises only resource-type of configurations. The +first option has the advantage of restoring a cluster to a clearly defined +state from a past point in time. The second option touches only security +configuration resources, but it is not a complete restore of {security-features}. + +To restore your security configuration from a backup, first make sure that the +repository holding `.security` snapshots is installed: -. Make sure the repository holding the `.security` snapshot is installed. -+ --- [source,js] -------------------------------------------------- GET /_snapshot/my_backup -------------------------------------------------- // CONSOLE --- +// TEST[continued] + +[source,js] +-------------------------------------------------- +GET /_snapshot/my_backup/snapshot_1 +-------------------------------------------------- +// CONSOLE +// TEST[continued] + +Then log into one of the node hosts, navigate to {es} installation directory, +and follow these steps: -. Add a new user with the `superuser` built-in role to the file based realm. +. Add a new user with the `superuser` built-in role to the +{stack-ov}/file-realm.html[file realm]. + -- -For example, create a user named `jacknich`: +For example, create a user named `restore_user`: [source,shell] -------------------------------------------------- -bin/elasticsearch-users useradd jacknich -p password -r superuser +bin/elasticsearch-users useradd restore_user -p password -r superuser -------------------------------------------------- -- @@ -304,14 +319,14 @@ bin/elasticsearch-users useradd jacknich -p password -r superuser -- [source,shell] -------------------------------------------------- -curl -u jacknich -X DELETE "localhost:9200/.security-7" +curl -u restore_user -X DELETE "localhost:9200/.security-*" -------------------------------------------------- WARNING: After this step any authentication that relies on the `.security` index will not work. This means that all API calls that authenticate with native or reserved users will fail, as will any user that relies on a native role. The file-realm user we created in the step above will continue to work -because it is not stored in the security index, and uses the builtin +because it is not stored in the `.security` index, and uses the builtin `superuser` role. -- @@ -321,33 +336,37 @@ because it is not stored in the security index, and uses the builtin -- [source,shell] -------------------------------------------------- - curl -u jacknich -X POST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -H 'Content-Type: application/json' -d' + curl -u restore_user -X POST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -H 'Content-Type: application/json' -d' { - "indices": ".security-7", - "include_global_state": true + "indices": ".security-*", + "include_global_state": true <1> } ' -------------------------------------------------- -NOTE: The `include_global_state` is optional but it will help to make sure the -configuration in the index is compatible with the rest of the configuration in -the cluster, as was the case at the time of the snapshot. But be advised that -this will also restore non-security persistent cluster settings. +<1> The `include_global_state: true` is mandatory only for a complete restore. +This will restore the global cluster metadata, which contains configurations +for the complete cluster. If you set this to `false` it will recover only the +contents of the `.security` index, i.e. usernames and password hashes, API +keys, application privileges, role and role mapping definitions. -- . Optionally, cherry-pick and <> that you have extracted with the `GET _cluster/settings` API, if you -need to review and override the settings that were included in the snapshot (by -the `include_global_state` flag). - -Lastly, if your backup included configuration files copy these and overwrite -the contents of `$ES_PATH_CONF` and restart the node. This needs to be done on -*every node*. Depending on the extent of the differences between your current -cluster configuration and the restored configuration, you may not be able to -perform a rolling restart. -If you are performing a full restore of your configuration directory, we recommend -a full cluster restart as the safest option. -Alternatively, you may wish to restore you configuration files to a separate -location on disk and use file comparison tools to review the differences between -your existing configuration and the restored configuration. -required. This is a conventional file configuration restore. +settings>> that you <> with the +`GET _cluster/settings` API, if you need to review and override the settings +that were included in the snapshot (by the `include_global_state` flag). + +If you pursue a complete point in time restore of the cluster, you also have to +restore configuration files. Again, this will restore non-security settings as +well. + +This entails a straight-up filesystem copy of the backed up configuration files +and overwrite of the contents of `$ES_PATH_CONF` and restart the node. This +needs to be done on *every node*. Depending on the extent of the differences +between your current cluster configuration and the restored configuration, you +may not be able to perform a rolling restart. If you are performing a full +restore of your configuration directory, we recommend a full cluster restart as +the safest option. Alternatively, you may wish to restore you configuration +files to a separate location on disk and use file comparison tools to review +the differences between your existing configuration and the restored +configuration. From d1fbeb4a42dd1c5eb56d1aca0974ce54d18fdfc5 Mon Sep 17 00:00:00 2001 From: lcawl Date: Thu, 27 Jun 2019 18:12:33 -0700 Subject: [PATCH 32/42] [DOCS] Reorganizes location of backup pages --- docs/reference/administering.asciidoc | 362 +----------------- .../backup-cluster-config.asciidoc | 59 +++ .../backup-cluster-data.asciidoc | 45 +++ .../administering/backup-cluster.asciidoc | 14 + .../backup-security-config.asciidoc | 269 +++++++++++++ docs/reference/modules/snapshots.asciidoc | 6 +- 6 files changed, 393 insertions(+), 362 deletions(-) create mode 100644 docs/reference/administering/backup-cluster-config.asciidoc create mode 100644 docs/reference/administering/backup-cluster-data.asciidoc create mode 100644 docs/reference/administering/backup-cluster.asciidoc create mode 100644 docs/reference/administering/backup-security-config.asciidoc diff --git a/docs/reference/administering.asciidoc b/docs/reference/administering.asciidoc index 2040da111f282..0a3901cf7ed30 100644 --- a/docs/reference/administering.asciidoc +++ b/docs/reference/administering.asciidoc @@ -9,364 +9,4 @@ cluster. -- -[[backup-cluster-data]] -== Back up a cluster's data - -As with any software that stores data, it is important to routinely back up your -data. {es} replicas provide high availability during runtime; they enable you to -tolerate sporadic node loss without an interruption of service. - -Replicas do not provide protection from catastrophic failure, however. For that, -you need a real backup of your cluster—a complete copy in case something goes -wrong. - -To back up your cluster's data, you can use the <>. - -include::{es-repo-dir}/modules/snapshots.asciidoc[tag=snapshot-intro] - -[role="xpack"] -[[backup-secured-cluster-data]] -== Data backup of a secured cluster - -In this context, a secured cluster designates a cluster with {es} -{security-features} enabled. Specifically, this means that the snapshot API -call needs to be authorized. - -`snapshot_user` is a reserved role that can be assigned to the user that the -snapshot endpoint is run as. This is the only role necessary if all the user -does is periodic snapshots as part of the backup procedure. Besides, it also -grants privileges to list all the existing snapshots (of any repository) as -well as list and view settings of all indices, including the `.security` index. -It does *not* grant privileges to create repositories, restore snapshots, or -search indices. Hence, the user can view and snapshot all indices, but cannot -access or modify any data. - -The restore API requires the `manage` cluster privilege. There is no bespoke -role for the restore process. This privilege is very permissive and should only -be granted to users in the "administrator" category. Specifically, it allows -malicious users to exfiltrate data to a location of their choosing. Automated -tools should not run as users with this privilege. - -For more information, see {stack-ov}/security-privileges.html[Security privileges] -and {stack-ov}/built-in-roles.html[Built-in roles]. - -[[backup-cluster-configuration]] -== Back up a cluster's configuration - -Besides data backup, configuration backup is also important, especially when it -becomes large and difficult to reconstruct. - -Configuration resides in <> on every -cluster node. Sensitive setting values, like for example passwords for -Watcher's notification servers, are specified inside a binary secure container, -the <> file. Moreover, some setting -values are file paths to the associated configuration data, for example the -ingest geo ip database. All these files are contained inside the `ES_PATH_CONF` -directory. - -NOTE: All changes to configuration files are done by manually editing the files -or using command line utilities, but *not* through APIs. In practice, these -changes are infrequent after the initial setup. - -We recommend that you take regular (ideally, daily) backups of your {es} config -(`$ES_PATH_CONF`) directory using the file backup software of your choice. -Because some of these files may contain sensitive data such as passwords and -TLS keys, you should investigate whether your backup software and/or storage -solution are able to encrypt this data. - -Using the <> it is possible to -override some settings from configuration files. You can capture these in a -*data* backup snapshot by specifing the `include_global_state: true` (default) -parameter to the snapshot API. Using the <> it is also possible to extract these configuration values in a text -format: - -[source,js] --------------------------------------------------- -GET _cluster/settings?pretty&flat_settings&filter_path=persistent --------------------------------------------------- -//CONSOLE -//TEST - -You can store the output of this as a file together with the rest of -configuration files. - -NOTE: Transient settings are not considered for backup. - -TIP: We recommend that you have a configuration management plan for these -configuration files. You may wish to check them into version control, or -provision them though your choice of configuration management tool. - -NOTE: {es} {security-features} store configuration data, like for example role -definitions and API keys, inside a dedicate special index. This "system" data, -complements the <> configuration and should -<>. - -NOTE: Other {stack} components, like Kibana and {ml-cap}, store their configuration -data inside other dedicated indices. From {es}'s perspective these are just data -so you can use the regular <> process. - -[role="xpack"] -[[security-backup]] -== Backup security configuration - -[role="xpack"] -[float] -[[backup-security-file-based-configuration]] -=== Backup file-based security configuration - -{es} {security-features} are configured using the <> inside the `elasticsearch.yml` and -`elasticsearch.keystore` files. In addition there are several other -<> inside the same `ES_PATH_CONF` -directory. These files define roles and role mappings as well as -<>. More, some of the -settings specify file paths to security sensitive data, such as TLS keys and -certificates for the HTTP client and inter-node communication and private key files for -<>, <> and the -<> realms. All these are also stored inside -`ES_PATH_CONF` (the path settings are relative). - -To back up all this configuration you can use a <>. - -NOTE: File backups must run on every cluster node. - -NOTE: File backups will store non-security configuration as well. Backing-up -only {security-features} configurations is not supported. A backup is a -point in time record of state of the complete configuration. - -IMPORTANT: The `elasticsearch.keystore`, TLS keys and SAML, OIDC, and Kerberos -realms private key files require confidentiality. This is crucial when files -are copied to the backup location, as this increases the surface for malicious -snooping. - -[role="xpack"] -[float] -[[backup-security-index-configuration]] -=== Backup security data - -{es} {security-features} store system configuration data inside a -dedicated index. This index is named `.security-6` in the {es} 6.x versions and -`.security-7` in the 7.x releases. The `.security` alias always points to the -appropriate index. This index contains the data which is not available in -configuration files and *cannot* be reliably backed up using standard -filesystem tools. This data describes: - -* the definition of users in the native realm (including hashed passwords) -* role definitions (defined via the <>) -* role mappings (defined via the <>) -* application privileges -* API keys - -Hence, `.security` contains resources and definitions more than configurations, -but all of which are required in a complete {security-features} backup. - -Use the <> to backup -`.security`, as you would for any <>. -For convenience, here are the complete steps: - -. Create a repository that you can use to backup the `.security` index. -It is preferable to have a <> for -this special index. - -+ --- -[source,js] ------------------------------------ -PUT /_snapshot/my_backup -{ - "type": "fs", - "settings": { - "location": "my_backup_location" - } -} ------------------------------------ -// CONSOLE - -The user calling this API must have the elevated `manage` cluster privilege, to -prevent <>. - --- - -. Create a user and assign it only the built-in `snapshot_user` role. -+ --- -The following example creates a new user `snapshot_user` in the -{stack-ov}/native-realm.html[native realm], but it is not important of which -realm the user is a member of: - -[source,js] --------------------------------------------------- -POST /_security/user/snapshot_user -{ - "password" : "secret", - "roles" : [ "snapshot_user" ] -} --------------------------------------------------- -// CONSOLE -// TEST[continued] - --- - -. Create incremental snapshots authorizing as `snapshot_user`. -+ --- -The following example shows how to use the create snapshot API to backup -the `.security` index to the `my_backup` repository created earlier. - -[source,js] --------------------------------------------------- -PUT /_snapshot/my_backup/snapshot_1 -{ - "indices": ".security", - "include_global_state": true <1> -} --------------------------------------------------- -// CONSOLE -// TEST[continued] - -<1> This captures all the persistent settings stored in the global cluster -metadata as well as other configurations such as aliases and stored scripts. -Note that this includes non-security configuration and that it complements but -does not replace the <>. - --- - -IMPORTANT: The index format is only compatible within a single major version, -and cannot be restored onto a version earlier that the version from which it -originated. e.g. You can restore a security snapshot from 6.6.0 into 6.7.0 -cluster, but you cannot restore it to a cluster running {es} 6.5.0 or 7.0.0. - -[float] -[[backup-security-repos]] -==== Controlling access to the backup repository - -The snapshot of the security index will typically contain sensitive data such -as user names and password hashes. Because password are stored using -<> the disclosure of a snapshot would -not automaticaly allow a third party to authenticate as one of your users or -use API keys, but it would disclose confidential information. - -It is also important that you protected the integrity of these backups in case -you ever need to restore them. If a third party is able to modify the stored -backups they may be able to install a back door that would grant access if the -snapshot is loaded into an {es} cluster. - -We recommend that you: - -* Snapshot the `.security` index in a dedicated repository, where read and write -access is strictly restricted and audited. -* If there are indications that the snapshot has been read, change the passwords -of the users in the native realm and revoke API keys. -* If there are indications that the snapshot has been tampered with, do not -restore it. There is currently no option for the restore process to detect -malicious tampering. - -[float] -[role="xpack"] -[[restore-security-configuration]] -=== Restoring security configuration - -NOTE: You can restore a snapshot of the `.security` index only if it was -created in a previous minor version in the same major version. The last minor -version of every major release can convert and read formats of the index for -both its major version and the next one. - -When you restore security configuration you have the option of doing a complete -restore of *all* configurations, including non-security ones, or to only restore -the contents of `.security`, which as <> described, comprises only resource-type of configurations. The -first option has the advantage of restoring a cluster to a clearly defined -state from a past point in time. The second option touches only security -configuration resources, but it is not a complete restore of {security-features}. - -To restore your security configuration from a backup, first make sure that the -repository holding `.security` snapshots is installed: - -[source,js] --------------------------------------------------- -GET /_snapshot/my_backup --------------------------------------------------- -// CONSOLE -// TEST[continued] - -[source,js] --------------------------------------------------- -GET /_snapshot/my_backup/snapshot_1 --------------------------------------------------- -// CONSOLE -// TEST[continued] - -Then log into one of the node hosts, navigate to {es} installation directory, -and follow these steps: - -. Add a new user with the `superuser` built-in role to the -{stack-ov}/file-realm.html[file realm]. -+ --- -For example, create a user named `restore_user`: -[source,shell] --------------------------------------------------- -bin/elasticsearch-users useradd restore_user -p password -r superuser --------------------------------------------------- --- - -. Using the previously created user, delete the existing `.security-6` or -`.security-7` index. -+ --- -[source,shell] --------------------------------------------------- -curl -u restore_user -X DELETE "localhost:9200/.security-*" --------------------------------------------------- - -WARNING: After this step any authentication that relies on the `.security` -index will not work. This means that all API calls that authenticate with -native or reserved users will fail, as will any user that relies on a native role. -The file-realm user we created in the step above will continue to work -because it is not stored in the `.security` index, and uses the builtin -`superuser` role. - --- - -. Using the same user, restore the `.security` index from the snapshot. -+ --- -[source,shell] --------------------------------------------------- - curl -u restore_user -X POST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -H 'Content-Type: application/json' -d' - { - "indices": ".security-*", - "include_global_state": true <1> - } - ' --------------------------------------------------- - -<1> The `include_global_state: true` is mandatory only for a complete restore. -This will restore the global cluster metadata, which contains configurations -for the complete cluster. If you set this to `false` it will recover only the -contents of the `.security` index, i.e. usernames and password hashes, API -keys, application privileges, role and role mapping definitions. --- - -. Optionally, cherry-pick and <> that you <> with the -`GET _cluster/settings` API, if you need to review and override the settings -that were included in the snapshot (by the `include_global_state` flag). - -If you pursue a complete point in time restore of the cluster, you also have to -restore configuration files. Again, this will restore non-security settings as -well. - -This entails a straight-up filesystem copy of the backed up configuration files -and overwrite of the contents of `$ES_PATH_CONF` and restart the node. This -needs to be done on *every node*. Depending on the extent of the differences -between your current cluster configuration and the restored configuration, you -may not be able to perform a rolling restart. If you are performing a full -restore of your configuration directory, we recommend a full cluster restart as -the safest option. Alternatively, you may wish to restore you configuration -files to a separate location on disk and use file comparison tools to review -the differences between your existing configuration and the restored -configuration. +include::administering/backup-cluster.asciidoc[] \ No newline at end of file diff --git a/docs/reference/administering/backup-cluster-config.asciidoc b/docs/reference/administering/backup-cluster-config.asciidoc new file mode 100644 index 0000000000000..0541c95b3fb97 --- /dev/null +++ b/docs/reference/administering/backup-cluster-config.asciidoc @@ -0,0 +1,59 @@ +[[backup-cluster-configuration]] +=== Back up a cluster's configuration +++++ +Back up the cluster configuration +++++ + +In addition to backing up the data in a cluster, it is important to back up its configuration--especially when the cluster becomes large and difficult to +reconstruct. + +Configuration information resides in +<> on every cluster node. Sensitive +setting values, like for example passwords for +Watcher's notification servers, are specified inside a binary secure container, +the <> file. Moreover, some setting +values are file paths to the associated configuration data, for example the +ingest geo ip database. All these files are contained inside the `ES_PATH_CONF` +directory. + +NOTE: All changes to configuration files are done by manually editing the files +or using command line utilities, but *not* through APIs. In practice, these +changes are infrequent after the initial setup. + +We recommend that you take regular (ideally, daily) backups of your {es} config +(`$ES_PATH_CONF`) directory using the file backup software of your choice. +Because some of these files may contain sensitive data such as passwords and +TLS keys, you should investigate whether your backup software and/or storage +solution are able to encrypt this data. + +Using the <> it is possible to +override some settings from configuration files. You can capture these in a +*data* backup snapshot by specifying the `include_global_state: true` (default) +parameter to the snapshot API. Using the +<> it is also possible to extract these +configuration values in a text format: + +[source,js] +-------------------------------------------------- +GET _cluster/settings?pretty&flat_settings&filter_path=persistent +-------------------------------------------------- +//CONSOLE +//TEST + +You can store the output of this as a file together with the rest of +configuration files. + +NOTE: Transient settings are not considered for backup. + +TIP: We recommend that you have a configuration management plan for these +configuration files. You may wish to check them into version control, or +provision them though your choice of configuration management tool. + +NOTE: {es} {security-features} store configuration data, like for example role +definitions and API keys, inside a dedicate special index. This "system" data, +complements the <> configuration and should +<>. + +NOTE: Other {stack} components, like Kibana and {ml-cap}, store their configuration +data inside other dedicated indices. From the {es} perspective these are just data +so you can use the regular <> process. diff --git a/docs/reference/administering/backup-cluster-data.asciidoc b/docs/reference/administering/backup-cluster-data.asciidoc new file mode 100644 index 0000000000000..f88ffb77c9bfe --- /dev/null +++ b/docs/reference/administering/backup-cluster-data.asciidoc @@ -0,0 +1,45 @@ +[[backup-cluster-data]] +=== Back up a cluster's data +++++ +Back up the data +++++ + +As with any software that stores data, it is important to routinely back up your +data. {es} replicas provide high availability during runtime; they enable you to +tolerate sporadic node loss without an interruption of service. + +Replicas do not provide protection from catastrophic failure, however. For that, +you need a real backup of your cluster—a complete copy in case something goes +wrong. + +To back up your cluster's data, you can use the <>. + +include::{es-repo-dir}/modules/snapshots.asciidoc[tag=snapshot-intro] + +[TIP] +==== +If your cluster has {es} {security-features} enabled, when you back up your data +the snapshot API call must be authorized. + +The `snapshot_user` role is a reserved role that can be assigned to the user +that the snapshot endpoint is run as. This is the only role necessary if all the user +does is periodic snapshots as part of the backup procedure. Besides, it also +grants privileges to list all the existing snapshots (of any repository) as +well as list and view settings of all indices, including the `.security` index. +It does *not* grant privileges to create repositories, restore snapshots, or +search indices. Hence, the user can view and snapshot all indices, but cannot +access or modify any data. + +For more information, see {stack-ov}/security-privileges.html[Security privileges] +and {stack-ov}/built-in-roles.html[Built-in roles]. +==== + +include::{es-repo-dir}/modules/snapshots.asciidoc[tag=restore-intro] + +[TIP] +==== +If your cluster has {es} {security-features} enabled, the restore API requires the `manage` cluster privilege. There is no bespoke role for the restore process. This privilege is very permissive and should only +be granted to users in the "administrator" category. Specifically, it allows +malicious users to exfiltrate data to a location of their choosing. Automated +tools should not run as users with this privilege. +==== diff --git a/docs/reference/administering/backup-cluster.asciidoc b/docs/reference/administering/backup-cluster.asciidoc new file mode 100644 index 0000000000000..5a2ca74b25e56 --- /dev/null +++ b/docs/reference/administering/backup-cluster.asciidoc @@ -0,0 +1,14 @@ +[[backup-cluster]] +== Back up a cluster + +include::{es-repo-dir}/modules/snapshots.asciidoc[tag=backup-warning] + +To have a complete backup for your cluster: + +. <> +. <> +. <> + +include::backup-cluster-data.asciidoc[] +include::backup-cluster-config.asciidoc[] +include::backup-security-config.asciidoc[] \ No newline at end of file diff --git a/docs/reference/administering/backup-security-config.asciidoc b/docs/reference/administering/backup-security-config.asciidoc new file mode 100644 index 0000000000000..096388548298e --- /dev/null +++ b/docs/reference/administering/backup-security-config.asciidoc @@ -0,0 +1,269 @@ +[role="xpack"] +[testenv="basic"] +[[security-backup]] +=== Back up a cluster's security configuration +++++ +Back up the security configuration +++++ + +Security configuration information resides in two places: +<> and +<>. + +[discrete] +[[backup-security-file-based-configuration]] +==== Back up file-based security configuration + +{es} {security-features} are configured using the <> inside the `elasticsearch.yml` and +`elasticsearch.keystore` files. In addition there are several other +<> inside the same `ES_PATH_CONF` +directory. These files define roles and role mappings as well as +<>. More, some of the +settings specify file paths to security sensitive data, such as TLS keys and +certificates for the HTTP client and inter-node communication and private key files for +<>, <> and the +<> realms. All these are also stored inside +`ES_PATH_CONF` (the path settings are relative). + +To back up all this configuration you can use a <>. + +NOTE: File backups must run on every cluster node. + +NOTE: File backups will store non-security configuration as well. Backing-up +only {security-features} configurations is not supported. A backup is a +point in time record of state of the complete configuration. + +IMPORTANT: The `elasticsearch.keystore`, TLS keys and SAML, OIDC, and Kerberos +realms private key files require confidentiality. This is crucial when files +are copied to the backup location, as this increases the surface for malicious +snooping. + +[discrete] +[[backup-security-index-configuration]] +==== Back up index-based security configuration + +{es} {security-features} store system configuration data inside a +dedicated index. This index is named `.security-6` in the {es} 6.x versions and +`.security-7` in the 7.x releases. The `.security` alias always points to the +appropriate index. This index contains the data which is not available in +configuration files and *cannot* be reliably backed up using standard +filesystem tools. This data describes: + +* the definition of users in the native realm (including hashed passwords) +* role definitions (defined via the <>) +* role mappings (defined via the <>) +* application privileges +* API keys + +Hence, `.security` contains resources and definitions more than configurations, +but all of which are required in a complete {security-features} backup. + +Use the <> to backup +`.security`, as you would for any <>. +For convenience, here are the complete steps: + +. Create a repository that you can use to backup the `.security` index. +It is preferable to have a <> for +this special index. + ++ +-- +[source,js] +----------------------------------- +PUT /_snapshot/my_backup +{ + "type": "fs", + "settings": { + "location": "my_backup_location" + } +} +----------------------------------- +// CONSOLE + +The user calling this API must have the elevated `manage` cluster privilege, to +prevent non-administrators exfiltrating data. + +-- + +. Create a user and assign it only the built-in `snapshot_user` role. ++ +-- +The following example creates a new user `snapshot_user` in the +{stack-ov}/native-realm.html[native realm], but it is not important of which +realm the user is a member of: + +[source,js] +-------------------------------------------------- +POST /_security/user/snapshot_user +{ + "password" : "secret", + "roles" : [ "snapshot_user" ] +} +-------------------------------------------------- +// CONSOLE +// TEST[continued] + +-- + +. Create incremental snapshots authorizing as `snapshot_user`. ++ +-- +The following example shows how to use the create snapshot API to backup +the `.security` index to the `my_backup` repository created earlier. + +[source,js] +-------------------------------------------------- +PUT /_snapshot/my_backup/snapshot_1 +{ + "indices": ".security", + "include_global_state": true <1> +} +-------------------------------------------------- +// CONSOLE +// TEST[continued] + +<1> This captures all the persistent settings stored in the global cluster +metadata as well as other configurations such as aliases and stored scripts. +Note that this includes non-security configuration and that it complements but +does not replace the <>. + +-- + +IMPORTANT: The index format is only compatible within a single major version, +and cannot be restored onto a version earlier that the version from which it +originated. e.g. You can restore a security snapshot from 6.6.0 into 6.7.0 +cluster, but you cannot restore it to a cluster running {es} 6.5.0 or 7.0.0. + +[discrete] +[[backup-security-repos]] +===== Controlling access to the backup repository + +The snapshot of the security index will typically contain sensitive data such +as user names and password hashes. Because password are stored using +<> the disclosure of a snapshot would +not automaticaly allow a third party to authenticate as one of your users or +use API keys, but it would disclose confidential information. + +It is also important that you protected the integrity of these backups in case +you ever need to restore them. If a third party is able to modify the stored +backups they may be able to install a back door that would grant access if the +snapshot is loaded into an {es} cluster. + +We recommend that you: + +* Snapshot the `.security` index in a dedicated repository, where read and write +access is strictly restricted and audited. +* If there are indications that the snapshot has been read, change the passwords +of the users in the native realm and revoke API keys. +* If there are indications that the snapshot has been tampered with, do not +restore it. There is currently no option for the restore process to detect +malicious tampering. + +[discrete] +[[restore-security-configuration]] +==== Restoring security configuration + +NOTE: You can restore a snapshot of the `.security` index only if it was +created in a previous minor version in the same major version. The last minor +version of every major release can convert and read formats of the index for +both its major version and the next one. + +When you restore security configuration you have the option of doing a complete +restore of *all* configurations, including non-security ones, or to only restore +the contents of `.security`, which as <> described, comprises only resource-type of configurations. The +first option has the advantage of restoring a cluster to a clearly defined +state from a past point in time. The second option touches only security +configuration resources, but it is not a complete restore of {security-features}. + +To restore your security configuration from a backup, first make sure that the +repository holding `.security` snapshots is installed: + +[source,js] +-------------------------------------------------- +GET /_snapshot/my_backup +-------------------------------------------------- +// CONSOLE +// TEST[continued] + +[source,js] +-------------------------------------------------- +GET /_snapshot/my_backup/snapshot_1 +-------------------------------------------------- +// CONSOLE +// TEST[continued] + +Then log into one of the node hosts, navigate to {es} installation directory, +and follow these steps: + +. Add a new user with the `superuser` built-in role to the +{stack-ov}/file-realm.html[file realm]. ++ +-- +For example, create a user named `restore_user`: +[source,shell] +-------------------------------------------------- +bin/elasticsearch-users useradd restore_user -p password -r superuser +-------------------------------------------------- +-- + +. Using the previously created user, delete the existing `.security-6` or +`.security-7` index. ++ +-- +[source,shell] +-------------------------------------------------- +curl -u restore_user -X DELETE "localhost:9200/.security-*" +-------------------------------------------------- + +WARNING: After this step any authentication that relies on the `.security` +index will not work. This means that all API calls that authenticate with +native or reserved users will fail, as will any user that relies on a native role. +The file-realm user we created in the step above will continue to work +because it is not stored in the `.security` index, and uses the builtin +`superuser` role. + +-- + +. Using the same user, restore the `.security` index from the snapshot. ++ +-- +[source,shell] +-------------------------------------------------- + curl -u restore_user -X POST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -H 'Content-Type: application/json' -d' + { + "indices": ".security-*", + "include_global_state": true <1> + } + ' +-------------------------------------------------- + +<1> The `include_global_state: true` is mandatory only for a complete restore. +This will restore the global cluster metadata, which contains configurations +for the complete cluster. If you set this to `false` it will recover only the +contents of the `.security` index, i.e. usernames and password hashes, API +keys, application privileges, role and role mapping definitions. +-- + +. Optionally, cherry-pick and <> that you <> with the +`GET _cluster/settings` API, if you need to review and override the settings +that were included in the snapshot (by the `include_global_state` flag). + +If you pursue a complete point in time restore of the cluster, you also have to +restore configuration files. Again, this will restore non-security settings as +well. + +This entails a straight-up filesystem copy of the backed up configuration files +and overwrite of the contents of `$ES_PATH_CONF` and restart the node. This +needs to be done on *every node*. Depending on the extent of the differences +between your current cluster configuration and the restored configuration, you +may not be able to perform a rolling restart. If you are performing a full +restore of your configuration directory, we recommend a full cluster restart as +the safest option. Alternatively, you may wish to restore you configuration +files to a separate location on disk and use file comparison tools to review +the differences between your existing configuration and the restored +configuration. diff --git a/docs/reference/modules/snapshots.asciidoc b/docs/reference/modules/snapshots.asciidoc index ed52c0958fec0..f78df39a60cd0 100644 --- a/docs/reference/modules/snapshots.asciidoc +++ b/docs/reference/modules/snapshots.asciidoc @@ -11,12 +11,16 @@ Snapshots are taken incrementally. This means that when it creates a snapshot of an index, Elasticsearch avoids copying any data that is already stored in the repository as part of an earlier snapshot of the same index. Therefore it can be efficient to take snapshots of your cluster quite frequently. +// end::snapshot-intro[] +// tag::restore-intro[] You can restore snapshots into a running cluster via the <>. When you restore an index, you can alter the name of the restored index as well as some of its settings. There is a great deal of flexibility in how the snapshot and restore functionality can be used. +// end::restore-intro[] +// tag::backup-warning[] WARNING: You cannot back up an Elasticsearch cluster by simply taking a copy of the data directories of all of its nodes. Elasticsearch may be making changes to the contents of its data directories while it is running; copying its data @@ -26,7 +30,7 @@ corruption and/or missing files. Alternatively, it may appear to have succeeded though it silently lost some of its data. The only reliable way to back up a cluster is by using the snapshot and restore functionality. -// end::snapshot-intro[] +// end::backup-warning[] [float] === Version compatibility From d997a8ae08c7bcc25d7b2833c8a84920a92a90e1 Mon Sep 17 00:00:00 2001 From: lcawl Date: Wed, 3 Jul 2019 18:16:06 -0700 Subject: [PATCH 33/42] [DOCS] Minor edits --- .../backup-cluster-config.asciidoc | 50 ++++++------- .../backup-security-config.asciidoc | 72 ++++++++++--------- 2 files changed, 64 insertions(+), 58 deletions(-) diff --git a/docs/reference/administering/backup-cluster-config.asciidoc b/docs/reference/administering/backup-cluster-config.asciidoc index 0541c95b3fb97..b60e7ae969cb1 100644 --- a/docs/reference/administering/backup-cluster-config.asciidoc +++ b/docs/reference/administering/backup-cluster-config.asciidoc @@ -9,12 +9,11 @@ reconstruct. Configuration information resides in <> on every cluster node. Sensitive -setting values, like for example passwords for -Watcher's notification servers, are specified inside a binary secure container, -the <> file. Moreover, some setting -values are file paths to the associated configuration data, for example the -ingest geo ip database. All these files are contained inside the `ES_PATH_CONF` -directory. +setting values such as passwords for the {watcher} notification servers, are +specified inside a binary secure container, the +<> file. Some setting values are +file paths to the associated configuration data, such as the ingest geo ip +database. All these files are contained inside the `ES_PATH_CONF` directory. NOTE: All changes to configuration files are done by manually editing the files or using command line utilities, but *not* through APIs. In practice, these @@ -22,16 +21,20 @@ changes are infrequent after the initial setup. We recommend that you take regular (ideally, daily) backups of your {es} config (`$ES_PATH_CONF`) directory using the file backup software of your choice. -Because some of these files may contain sensitive data such as passwords and -TLS keys, you should investigate whether your backup software and/or storage +Some of these files may contain sensitive data such as passwords and TLS keys, +therefore you should investigate whether your backup software and/or storage solution are able to encrypt this data. -Using the <> it is possible to -override some settings from configuration files. You can capture these in a -*data* backup snapshot by specifying the `include_global_state: true` (default) -parameter to the snapshot API. Using the -<> it is also possible to extract these -configuration values in a text format: +TIP: We recommend that you have a configuration management plan for these +configuration files. You may wish to check them into version control, or +provision them though your choice of configuration management tool. + +Some settings in configuration files might be overridden by +<>. You can capture these settings in +a *data* backup snapshot by specifying the `include_global_state: true` (default) +parameter for the snapshot API. Alternatively, you can extract these +configuration values in text format by using the +<>: [source,js] -------------------------------------------------- @@ -43,17 +46,16 @@ GET _cluster/settings?pretty&flat_settings&filter_path=persistent You can store the output of this as a file together with the rest of configuration files. -NOTE: Transient settings are not considered for backup. - -TIP: We recommend that you have a configuration management plan for these -configuration files. You may wish to check them into version control, or -provision them though your choice of configuration management tool. +[NOTE] +==== -NOTE: {es} {security-features} store configuration data, like for example role -definitions and API keys, inside a dedicate special index. This "system" data, +* Transient settings are not considered for backup. +* {es} {security-features} store configuration data such as role definitions and +API keys inside a dedicate special index. This "system" data, complements the <> configuration and should -<>. - -NOTE: Other {stack} components, like Kibana and {ml-cap}, store their configuration +be <>. +* Other {stack} components, like Kibana and {ml-cap}, store their configuration data inside other dedicated indices. From the {es} perspective these are just data so you can use the regular <> process. + +==== diff --git a/docs/reference/administering/backup-security-config.asciidoc b/docs/reference/administering/backup-security-config.asciidoc index 096388548298e..9466cd0a2091a 100644 --- a/docs/reference/administering/backup-security-config.asciidoc +++ b/docs/reference/administering/backup-security-config.asciidoc @@ -18,27 +18,31 @@ Security configuration information resides in two places: `xpack.security` namespace>> inside the `elasticsearch.yml` and `elasticsearch.keystore` files. In addition there are several other <> inside the same `ES_PATH_CONF` -directory. These files define roles and role mappings as well as -<>. More, some of the -settings specify file paths to security sensitive data, such as TLS keys and +directory. These files define roles and role mappings and +<>. Some of the +settings specify file paths to security-sensitive data, such as TLS keys and certificates for the HTTP client and inter-node communication and private key files for <>, <> and the <> realms. All these are also stored inside -`ES_PATH_CONF` (the path settings are relative). +`ES_PATH_CONF`;the path settings are relative. + +IMPORTANT: The `elasticsearch.keystore`, TLS keys and SAML, OIDC, and Kerberos +realms private key files require confidentiality. This is crucial when files +are copied to the backup location, as this increases the surface for malicious +snooping. To back up all this configuration you can use a <>. -NOTE: File backups must run on every cluster node. +[NOTE] +==== -NOTE: File backups will store non-security configuration as well. Backing-up -only {security-features} configurations is not supported. A backup is a +* File backups must run on every cluster node. +* File backups will store non-security configuration as well. Backing-up +only {security-features} configuration is not supported. A backup is a point in time record of state of the complete configuration. -IMPORTANT: The `elasticsearch.keystore`, TLS keys and SAML, OIDC, and Kerberos -realms private key files require confidentiality. This is crucial when files -are copied to the backup location, as this increases the surface for malicious -snooping. +==== [discrete] [[backup-security-index-configuration]] @@ -52,13 +56,15 @@ configuration files and *cannot* be reliably backed up using standard filesystem tools. This data describes: * the definition of users in the native realm (including hashed passwords) -* role definitions (defined via the <>) -* role mappings (defined via the <>) +* role definitions (defined via the <>) +* role mappings (defined via the + <>) * application privileges * API keys -Hence, `.security` contains resources and definitions more than configurations, -but all of which are required in a complete {security-features} backup. +The `.security` index thus contains resources and definitions in addition to +configuration information. All of that information is required in a complete +{security-features} backup. Use the <> to backup `.security`, as you would for any <>. @@ -67,7 +73,6 @@ For convenience, here are the complete steps: . Create a repository that you can use to backup the `.security` index. It is preferable to have a <> for this special index. - + -- [source,js] @@ -82,7 +87,7 @@ PUT /_snapshot/my_backup ----------------------------------- // CONSOLE -The user calling this API must have the elevated `manage` cluster privilege, to +The user calling this API must have the elevated `manage` cluster privilege to prevent non-administrators exfiltrating data. -- @@ -91,7 +96,7 @@ prevent non-administrators exfiltrating data. + -- The following example creates a new user `snapshot_user` in the -{stack-ov}/native-realm.html[native realm], but it is not important of which +{stack-ov}/native-realm.html[native realm], but it is not important which realm the user is a member of: [source,js] @@ -107,11 +112,11 @@ POST /_security/user/snapshot_user -- -. Create incremental snapshots authorizing as `snapshot_user`. +. Create incremental snapshots authorized as `snapshot_user`. + -- The following example shows how to use the create snapshot API to backup -the `.security` index to the `my_backup` repository created earlier. +the `.security` index to the `my_backup` repository: [source,js] -------------------------------------------------- @@ -124,32 +129,31 @@ PUT /_snapshot/my_backup/snapshot_1 // CONSOLE // TEST[continued] -<1> This captures all the persistent settings stored in the global cluster -metadata as well as other configurations such as aliases and stored scripts. -Note that this includes non-security configuration and that it complements but -does not replace the <>. +<1> This parameter value captures all the persistent settings stored in the +global cluster metadata as well as other configurations such as aliases and +stored scripts. Note that this includes non-security configuration and that it complements but does not replace the +<>. -- IMPORTANT: The index format is only compatible within a single major version, -and cannot be restored onto a version earlier that the version from which it -originated. e.g. You can restore a security snapshot from 6.6.0 into 6.7.0 -cluster, but you cannot restore it to a cluster running {es} 6.5.0 or 7.0.0. +and cannot be restored onto a version earlier than the version from which it +originated. For exmaple, you can restore a security snapshot from 6.6.0 into a +6.7.0 cluster, but you cannot restore it to a cluster running {es} 6.5.0 or 7.0.0. [discrete] [[backup-security-repos]] ===== Controlling access to the backup repository The snapshot of the security index will typically contain sensitive data such -as user names and password hashes. Because password are stored using -<> the disclosure of a snapshot would -not automaticaly allow a third party to authenticate as one of your users or -use API keys, but it would disclose confidential information. +as user names and password hashes. Because passwords are stored using +<>, the disclosure of a snapshot would +not automatically enable a third party to authenticate as one of your users or +use API keys. However, it would disclose confidential information. -It is also important that you protected the integrity of these backups in case +It is also important that you protect the integrity of these backups in case you ever need to restore them. If a third party is able to modify the stored -backups they may be able to install a back door that would grant access if the +backups, they may be able to install a back door that would grant access if the snapshot is loaded into an {es} cluster. We recommend that you: From a037fdf30bdeafad25ca19163966c38e165b3afa Mon Sep 17 00:00:00 2001 From: lcawl Date: Wed, 3 Jul 2019 18:35:35 -0700 Subject: [PATCH 34/42] [DOCS] Splits separate restore pages --- .../backup-cluster-data.asciidoc | 10 -- .../administering/backup-cluster.asciidoc | 11 +- .../backup-security-config.asciidoc | 106 ---------------- .../restore-cluster-data.asciidoc | 15 +++ .../restore-security-config.asciidoc | 113 ++++++++++++++++++ 5 files changed, 138 insertions(+), 117 deletions(-) create mode 100644 docs/reference/administering/restore-cluster-data.asciidoc create mode 100644 docs/reference/administering/restore-security-config.asciidoc diff --git a/docs/reference/administering/backup-cluster-data.asciidoc b/docs/reference/administering/backup-cluster-data.asciidoc index f88ffb77c9bfe..b84d9fb56abfd 100644 --- a/docs/reference/administering/backup-cluster-data.asciidoc +++ b/docs/reference/administering/backup-cluster-data.asciidoc @@ -33,13 +33,3 @@ access or modify any data. For more information, see {stack-ov}/security-privileges.html[Security privileges] and {stack-ov}/built-in-roles.html[Built-in roles]. ==== - -include::{es-repo-dir}/modules/snapshots.asciidoc[tag=restore-intro] - -[TIP] -==== -If your cluster has {es} {security-features} enabled, the restore API requires the `manage` cluster privilege. There is no bespoke role for the restore process. This privilege is very permissive and should only -be granted to users in the "administrator" category. Specifically, it allows -malicious users to exfiltrate data to a location of their choosing. Automated -tools should not run as users with this privilege. -==== diff --git a/docs/reference/administering/backup-cluster.asciidoc b/docs/reference/administering/backup-cluster.asciidoc index 5a2ca74b25e56..facf8402d39a0 100644 --- a/docs/reference/administering/backup-cluster.asciidoc +++ b/docs/reference/administering/backup-cluster.asciidoc @@ -9,6 +9,15 @@ To have a complete backup for your cluster: . <> . <> +To restore your cluster from a backup: + +. <> +. <> + + + include::backup-cluster-data.asciidoc[] include::backup-cluster-config.asciidoc[] -include::backup-security-config.asciidoc[] \ No newline at end of file +include::backup-security-config.asciidoc[] +include::restore-cluster-data.asciidoc[] +include::restore-security-config.asciidoc[] \ No newline at end of file diff --git a/docs/reference/administering/backup-security-config.asciidoc b/docs/reference/administering/backup-security-config.asciidoc index 9466cd0a2091a..b2caa248197b8 100644 --- a/docs/reference/administering/backup-security-config.asciidoc +++ b/docs/reference/administering/backup-security-config.asciidoc @@ -165,109 +165,3 @@ of the users in the native realm and revoke API keys. * If there are indications that the snapshot has been tampered with, do not restore it. There is currently no option for the restore process to detect malicious tampering. - -[discrete] -[[restore-security-configuration]] -==== Restoring security configuration - -NOTE: You can restore a snapshot of the `.security` index only if it was -created in a previous minor version in the same major version. The last minor -version of every major release can convert and read formats of the index for -both its major version and the next one. - -When you restore security configuration you have the option of doing a complete -restore of *all* configurations, including non-security ones, or to only restore -the contents of `.security`, which as <> described, comprises only resource-type of configurations. The -first option has the advantage of restoring a cluster to a clearly defined -state from a past point in time. The second option touches only security -configuration resources, but it is not a complete restore of {security-features}. - -To restore your security configuration from a backup, first make sure that the -repository holding `.security` snapshots is installed: - -[source,js] --------------------------------------------------- -GET /_snapshot/my_backup --------------------------------------------------- -// CONSOLE -// TEST[continued] - -[source,js] --------------------------------------------------- -GET /_snapshot/my_backup/snapshot_1 --------------------------------------------------- -// CONSOLE -// TEST[continued] - -Then log into one of the node hosts, navigate to {es} installation directory, -and follow these steps: - -. Add a new user with the `superuser` built-in role to the -{stack-ov}/file-realm.html[file realm]. -+ --- -For example, create a user named `restore_user`: -[source,shell] --------------------------------------------------- -bin/elasticsearch-users useradd restore_user -p password -r superuser --------------------------------------------------- --- - -. Using the previously created user, delete the existing `.security-6` or -`.security-7` index. -+ --- -[source,shell] --------------------------------------------------- -curl -u restore_user -X DELETE "localhost:9200/.security-*" --------------------------------------------------- - -WARNING: After this step any authentication that relies on the `.security` -index will not work. This means that all API calls that authenticate with -native or reserved users will fail, as will any user that relies on a native role. -The file-realm user we created in the step above will continue to work -because it is not stored in the `.security` index, and uses the builtin -`superuser` role. - --- - -. Using the same user, restore the `.security` index from the snapshot. -+ --- -[source,shell] --------------------------------------------------- - curl -u restore_user -X POST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -H 'Content-Type: application/json' -d' - { - "indices": ".security-*", - "include_global_state": true <1> - } - ' --------------------------------------------------- - -<1> The `include_global_state: true` is mandatory only for a complete restore. -This will restore the global cluster metadata, which contains configurations -for the complete cluster. If you set this to `false` it will recover only the -contents of the `.security` index, i.e. usernames and password hashes, API -keys, application privileges, role and role mapping definitions. --- - -. Optionally, cherry-pick and <> that you <> with the -`GET _cluster/settings` API, if you need to review and override the settings -that were included in the snapshot (by the `include_global_state` flag). - -If you pursue a complete point in time restore of the cluster, you also have to -restore configuration files. Again, this will restore non-security settings as -well. - -This entails a straight-up filesystem copy of the backed up configuration files -and overwrite of the contents of `$ES_PATH_CONF` and restart the node. This -needs to be done on *every node*. Depending on the extent of the differences -between your current cluster configuration and the restored configuration, you -may not be able to perform a rolling restart. If you are performing a full -restore of your configuration directory, we recommend a full cluster restart as -the safest option. Alternatively, you may wish to restore you configuration -files to a separate location on disk and use file comparison tools to review -the differences between your existing configuration and the restored -configuration. diff --git a/docs/reference/administering/restore-cluster-data.asciidoc b/docs/reference/administering/restore-cluster-data.asciidoc new file mode 100644 index 0000000000000..c9ae6da339fd4 --- /dev/null +++ b/docs/reference/administering/restore-cluster-data.asciidoc @@ -0,0 +1,15 @@ +[[restore-cluster-data]] +=== Restore a cluster's data +++++ +Restore the data +++++ + +include::{es-repo-dir}/modules/snapshots.asciidoc[tag=restore-intro] + +[TIP] +==== +If your cluster has {es} {security-features} enabled, the restore API requires the `manage` cluster privilege. There is no bespoke role for the restore process. This privilege is very permissive and should only +be granted to users in the "administrator" category. Specifically, it allows +malicious users to exfiltrate data to a location of their choosing. Automated +tools should not run as users with this privilege. +==== diff --git a/docs/reference/administering/restore-security-config.asciidoc b/docs/reference/administering/restore-security-config.asciidoc new file mode 100644 index 0000000000000..d0a8301d55416 --- /dev/null +++ b/docs/reference/administering/restore-security-config.asciidoc @@ -0,0 +1,113 @@ +[role="xpack"] +[testenv="basic"] +[[restore-security-configuration]] +=== Restore a cluster's security configuration +++++ +Restore the security configuration +++++ + +NOTE: You can restore a snapshot of the `.security` index only if it was +created in a previous minor version in the same major version. The last minor +version of every major release can convert and read formats of the index for +both its major version and the next one. + +When you restore security configuration you have the option of doing a complete +restore of *all* configurations, including non-security ones, or to only restore +the contents of the `.security` index. As described in +<>, the second option comprises only +resource-type configurations. The first option has the advantage of restoring +a cluster to a clearly defined state from a past point in time. The second option +touches only security configuration resources, but it does not completely restore +the {security-features}. + +To restore your security configuration from a backup, first make sure that the +repository holding `.security` snapshots is installed: + +[source,js] +-------------------------------------------------- +GET /_snapshot/my_backup +-------------------------------------------------- +// CONSOLE +// TEST[continued] + +[source,js] +-------------------------------------------------- +GET /_snapshot/my_backup/snapshot_1 +-------------------------------------------------- +// CONSOLE +// TEST[continued] + +Then log into one of the node hosts, navigate to {es} installation directory, +and follow these steps: + +. Add a new user with the `superuser` built-in role to the +{stack-ov}/file-realm.html[file realm]. ++ +-- +For example, create a user named `restore_user`: +[source,shell] +-------------------------------------------------- +bin/elasticsearch-users useradd restore_user -p password -r superuser +-------------------------------------------------- +-- + +. Using the previously created user, delete the existing `.security-6` or +`.security-7` index. ++ +-- +[source,shell] +-------------------------------------------------- +curl -u restore_user -X DELETE "localhost:9200/.security-*" +-------------------------------------------------- + +WARNING: After this step any authentication that relies on the `.security` +index will not work. This means that all API calls that authenticate with +native or reserved users will fail, as will any user that relies on a native role. +The file realm user we created in the step above will continue to work +because it is not stored in the `.security` index and uses the built-in +`superuser` role. + +-- + +. Using the same user, restore the `.security` index from the snapshot. ++ +-- +[source,shell] +-------------------------------------------------- + curl -u restore_user -X POST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -H 'Content-Type: application/json' -d' + { + "indices": ".security-*", + "include_global_state": true <1> + } + ' +-------------------------------------------------- + +<1> The `include_global_state: true` is mandatory only for a complete restore. +This will restore the global cluster metadata, which contains configuration +information for the complete cluster. If you set this to `false`, it recovers +only the contents of the `.security` index, such as usernames and password +hashes, API keys, application privileges, role and role mapping definitions. +-- + +. Optionally, if you need to review and override the settings that were included +in the snapshot (by the `include_global_state` flag), cherry-pick and +<> that you +<> with the +`GET _cluster/settings` API. + +. If you pursue a complete point in time restore of the cluster, you also have +to restore configuration files. Again, this will restore non-security settings as +well. ++ +-- +This entails a straight-up filesystem copy of the backed up configuration files, +overwriting the contents of `$ES_PATH_CONF`, and restarting the node. This +needs to be done on *every node*. Depending on the extent of the differences +between your current cluster configuration and the restored configuration, you +may not be able to perform a rolling restart. If you are performing a full +restore of your configuration directory, we recommend a full cluster restart as +the safest option. Alternatively, you may wish to restore your configuration +files to a separate location on disk and use file comparison tools to review +the differences between your existing configuration and the restored +configuration. +-- \ No newline at end of file From c87b02e9d3c42f973aea345e6e914231666bbbb9 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Wed, 10 Jul 2019 00:00:51 +0300 Subject: [PATCH 35/42] Update docs/reference/administering/backup-cluster-data.asciidoc Co-Authored-By: Tim Vernum --- docs/reference/administering/backup-cluster-data.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/administering/backup-cluster-data.asciidoc b/docs/reference/administering/backup-cluster-data.asciidoc index b84d9fb56abfd..bc77138cf75e1 100644 --- a/docs/reference/administering/backup-cluster-data.asciidoc +++ b/docs/reference/administering/backup-cluster-data.asciidoc @@ -22,7 +22,7 @@ If your cluster has {es} {security-features} enabled, when you back up your data the snapshot API call must be authorized. The `snapshot_user` role is a reserved role that can be assigned to the user -that the snapshot endpoint is run as. This is the only role necessary if all the user +who is calling the snapshot endpoint. This is the only role necessary if all the user does is periodic snapshots as part of the backup procedure. Besides, it also grants privileges to list all the existing snapshots (of any repository) as well as list and view settings of all indices, including the `.security` index. From 32b880b9a10b6dcba50b8a62df386819eac95a0a Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Wed, 10 Jul 2019 00:04:09 +0300 Subject: [PATCH 36/42] Update docs/reference/administering/backup-cluster-data.asciidoc Co-Authored-By: Tim Vernum --- docs/reference/administering/backup-cluster-data.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/administering/backup-cluster-data.asciidoc b/docs/reference/administering/backup-cluster-data.asciidoc index bc77138cf75e1..0515cf7baa1b3 100644 --- a/docs/reference/administering/backup-cluster-data.asciidoc +++ b/docs/reference/administering/backup-cluster-data.asciidoc @@ -27,7 +27,7 @@ does is periodic snapshots as part of the backup procedure. Besides, it also grants privileges to list all the existing snapshots (of any repository) as well as list and view settings of all indices, including the `.security` index. It does *not* grant privileges to create repositories, restore snapshots, or -search indices. Hence, the user can view and snapshot all indices, but cannot +search within indices. Hence, the user can view and snapshot all indices, but cannot access or modify any data. For more information, see {stack-ov}/security-privileges.html[Security privileges] From 7a5912e8558bcfa21844d677e66cbc04bcf00e25 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Wed, 10 Jul 2019 00:05:38 +0300 Subject: [PATCH 37/42] Update docs/reference/administering/backup-security-config.asciidoc Co-Authored-By: Tim Vernum --- docs/reference/administering/backup-security-config.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/administering/backup-security-config.asciidoc b/docs/reference/administering/backup-security-config.asciidoc index b2caa248197b8..cb5771cc2769f 100644 --- a/docs/reference/administering/backup-security-config.asciidoc +++ b/docs/reference/administering/backup-security-config.asciidoc @@ -24,7 +24,7 @@ settings specify file paths to security-sensitive data, such as TLS keys and certificates for the HTTP client and inter-node communication and private key files for <>, <> and the <> realms. All these are also stored inside -`ES_PATH_CONF`;the path settings are relative. +`ES_PATH_CONF`; the path settings are relative. IMPORTANT: The `elasticsearch.keystore`, TLS keys and SAML, OIDC, and Kerberos realms private key files require confidentiality. This is crucial when files From ecae90e7b8e93924d716405340a7a317c34ef456 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Wed, 10 Jul 2019 00:06:44 +0300 Subject: [PATCH 38/42] Update docs/reference/administering/backup-security-config.asciidoc Co-Authored-By: Tim Vernum --- docs/reference/administering/backup-security-config.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/administering/backup-security-config.asciidoc b/docs/reference/administering/backup-security-config.asciidoc index cb5771cc2769f..432ad5fb665e6 100644 --- a/docs/reference/administering/backup-security-config.asciidoc +++ b/docs/reference/administering/backup-security-config.asciidoc @@ -138,7 +138,7 @@ stored scripts. Note that this includes non-security configuration and that it c IMPORTANT: The index format is only compatible within a single major version, and cannot be restored onto a version earlier than the version from which it -originated. For exmaple, you can restore a security snapshot from 6.6.0 into a +originated. For example, you can restore a security snapshot from 6.6.0 into a 6.7.0 cluster, but you cannot restore it to a cluster running {es} 6.5.0 or 7.0.0. [discrete] From 2f72866429c96fed14be7a7853072de0c3e6093c Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Wed, 10 Jul 2019 00:11:31 +0300 Subject: [PATCH 39/42] Update docs/reference/administering/backup-security-config.asciidoc Co-Authored-By: Tim Vernum --- docs/reference/administering/backup-security-config.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/administering/backup-security-config.asciidoc b/docs/reference/administering/backup-security-config.asciidoc index 432ad5fb665e6..c835d0ba58012 100644 --- a/docs/reference/administering/backup-security-config.asciidoc +++ b/docs/reference/administering/backup-security-config.asciidoc @@ -72,7 +72,7 @@ For convenience, here are the complete steps: . Create a repository that you can use to backup the `.security` index. It is preferable to have a <> for -this special index. +this special index. If you wish, you can also snapshot the system indices for other {stack} components to this repository. + -- [source,js] From 34dc16b30ce1b665fafb87363451f727f870fa6a Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Wed, 10 Jul 2019 00:36:54 +0300 Subject: [PATCH 40/42] More reviews --- .../reference/administering/backup-cluster-config.asciidoc | 7 ++++--- docs/reference/administering/backup-cluster-data.asciidoc | 4 ++-- .../administering/backup-security-config.asciidoc | 2 +- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/reference/administering/backup-cluster-config.asciidoc b/docs/reference/administering/backup-cluster-config.asciidoc index b60e7ae969cb1..373ff48618de8 100644 --- a/docs/reference/administering/backup-cluster-config.asciidoc +++ b/docs/reference/administering/backup-cluster-config.asciidoc @@ -21,14 +21,15 @@ changes are infrequent after the initial setup. We recommend that you take regular (ideally, daily) backups of your {es} config (`$ES_PATH_CONF`) directory using the file backup software of your choice. -Some of these files may contain sensitive data such as passwords and TLS keys, -therefore you should investigate whether your backup software and/or storage -solution are able to encrypt this data. TIP: We recommend that you have a configuration management plan for these configuration files. You may wish to check them into version control, or provision them though your choice of configuration management tool. +Some of these files may contain sensitive data such as passwords and TLS keys, +therefore you should investigate whether your backup software and/or storage +solution are able to encrypt this data. + Some settings in configuration files might be overridden by <>. You can capture these settings in a *data* backup snapshot by specifying the `include_global_state: true` (default) diff --git a/docs/reference/administering/backup-cluster-data.asciidoc b/docs/reference/administering/backup-cluster-data.asciidoc index 0515cf7baa1b3..063018337d666 100644 --- a/docs/reference/administering/backup-cluster-data.asciidoc +++ b/docs/reference/administering/backup-cluster-data.asciidoc @@ -23,8 +23,8 @@ the snapshot API call must be authorized. The `snapshot_user` role is a reserved role that can be assigned to the user who is calling the snapshot endpoint. This is the only role necessary if all the user -does is periodic snapshots as part of the backup procedure. Besides, it also -grants privileges to list all the existing snapshots (of any repository) as +does is periodic snapshots as part of the backup procedure. This role includes +the privileges to list all the existing snapshots (of any repository) as well as list and view settings of all indices, including the `.security` index. It does *not* grant privileges to create repositories, restore snapshots, or search within indices. Hence, the user can view and snapshot all indices, but cannot diff --git a/docs/reference/administering/backup-security-config.asciidoc b/docs/reference/administering/backup-security-config.asciidoc index c835d0ba58012..b60d2211b98b4 100644 --- a/docs/reference/administering/backup-security-config.asciidoc +++ b/docs/reference/administering/backup-security-config.asciidoc @@ -32,7 +32,7 @@ are copied to the backup location, as this increases the surface for malicious snooping. To back up all this configuration you can use a <>. +conventional file-based backup>>, as described in the previous section. [NOTE] ==== From a4454e78acd1922fbd548e7f01bbd6a5d7d13163 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Wed, 10 Jul 2019 00:45:01 +0300 Subject: [PATCH 41/42] Mark curl sh cmds as NOTCONSOLE --- .../reference/administering/restore-security-config.asciidoc | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/reference/administering/restore-security-config.asciidoc b/docs/reference/administering/restore-security-config.asciidoc index d0a8301d55416..70ccb8530cf36 100644 --- a/docs/reference/administering/restore-security-config.asciidoc +++ b/docs/reference/administering/restore-security-config.asciidoc @@ -28,7 +28,6 @@ repository holding `.security` snapshots is installed: GET /_snapshot/my_backup -------------------------------------------------- // CONSOLE -// TEST[continued] [source,js] -------------------------------------------------- @@ -59,6 +58,7 @@ bin/elasticsearch-users useradd restore_user -p password -r superuser -------------------------------------------------- curl -u restore_user -X DELETE "localhost:9200/.security-*" -------------------------------------------------- +// NOTCONSOLE WARNING: After this step any authentication that relies on the `.security` index will not work. This means that all API calls that authenticate with @@ -81,6 +81,7 @@ because it is not stored in the `.security` index and uses the built-in } ' -------------------------------------------------- +// NOTCONSOLE <1> The `include_global_state: true` is mandatory only for a complete restore. This will restore the global cluster metadata, which contains configuration @@ -110,4 +111,4 @@ the safest option. Alternatively, you may wish to restore your configuration files to a separate location on disk and use file comparison tools to review the differences between your existing configuration and the restored configuration. --- \ No newline at end of file +-- From 39c5952f7f6a780efc3e51bb2e869d8e9b91aeb6 Mon Sep 17 00:00:00 2001 From: Albert Zaharovits Date: Wed, 10 Jul 2019 01:36:17 +0300 Subject: [PATCH 42/42] merge backup and restore files --- ...ckup-and-restore-security-config.asciidoc} | 118 +++++++++++++++++- .../administering/backup-cluster.asciidoc | 3 +- .../restore-security-config.asciidoc | 114 ----------------- 3 files changed, 117 insertions(+), 118 deletions(-) rename docs/reference/administering/{backup-security-config.asciidoc => backup-and-restore-security-config.asciidoc} (58%) delete mode 100644 docs/reference/administering/restore-security-config.asciidoc diff --git a/docs/reference/administering/backup-security-config.asciidoc b/docs/reference/administering/backup-and-restore-security-config.asciidoc similarity index 58% rename from docs/reference/administering/backup-security-config.asciidoc rename to docs/reference/administering/backup-and-restore-security-config.asciidoc index b60d2211b98b4..847557db48610 100644 --- a/docs/reference/administering/backup-security-config.asciidoc +++ b/docs/reference/administering/backup-and-restore-security-config.asciidoc @@ -108,7 +108,7 @@ POST /_security/user/snapshot_user } -------------------------------------------------- // CONSOLE -// TEST[continued] +// TEST[skip:security is not enabled in this fixture] -- @@ -138,7 +138,7 @@ stored scripts. Note that this includes non-security configuration and that it c IMPORTANT: The index format is only compatible within a single major version, and cannot be restored onto a version earlier than the version from which it -originated. For example, you can restore a security snapshot from 6.6.0 into a +originated. For example, you can restore a security snapshot from 6.6.0 into a 6.7.0 cluster, but you cannot restore it to a cluster running {es} 6.5.0 or 7.0.0. [discrete] @@ -165,3 +165,117 @@ of the users in the native realm and revoke API keys. * If there are indications that the snapshot has been tampered with, do not restore it. There is currently no option for the restore process to detect malicious tampering. + +[[restore-security-configuration]] +=== Restore a cluster's security configuration +++++ +Restore the security configuration +++++ + +NOTE: You can restore a snapshot of the `.security` index only if it was +created in a previous minor version in the same major version. The last minor +version of every major release can convert and read formats of the index for +both its major version and the next one. + +When you restore security configuration you have the option of doing a complete +restore of *all* configurations, including non-security ones, or to only restore +the contents of the `.security` index. As described in +<>, the second option comprises only +resource-type configurations. The first option has the advantage of restoring +a cluster to a clearly defined state from a past point in time. The second option +touches only security configuration resources, but it does not completely restore +the {security-features}. + +To restore your security configuration from a backup, first make sure that the +repository holding `.security` snapshots is installed: + +[source,js] +-------------------------------------------------- +GET /_snapshot/my_backup +-------------------------------------------------- +// CONSOLE +// TEST[continued] + +[source,js] +-------------------------------------------------- +GET /_snapshot/my_backup/snapshot_1 +-------------------------------------------------- +// CONSOLE +// TEST[continued] + +Then log into one of the node hosts, navigate to {es} installation directory, +and follow these steps: + +. Add a new user with the `superuser` built-in role to the +{stack-ov}/file-realm.html[file realm]. ++ +-- +For example, create a user named `restore_user`: +[source,shell] +-------------------------------------------------- +bin/elasticsearch-users useradd restore_user -p password -r superuser +-------------------------------------------------- +-- + +. Using the previously created user, delete the existing `.security-6` or +`.security-7` index. ++ +-- +[source,shell] +-------------------------------------------------- +curl -u restore_user -X DELETE "localhost:9200/.security-*" +-------------------------------------------------- +// NOTCONSOLE + +WARNING: After this step any authentication that relies on the `.security` +index will not work. This means that all API calls that authenticate with +native or reserved users will fail, as will any user that relies on a native role. +The file realm user we created in the step above will continue to work +because it is not stored in the `.security` index and uses the built-in +`superuser` role. + +-- + +. Using the same user, restore the `.security` index from the snapshot. ++ +-- +[source,shell] +-------------------------------------------------- + curl -u restore_user -X POST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -H 'Content-Type: application/json' -d' + { + "indices": ".security-*", + "include_global_state": true <1> + } + ' +-------------------------------------------------- +// NOTCONSOLE + +<1> The `include_global_state: true` is mandatory only for a complete restore. +This will restore the global cluster metadata, which contains configuration +information for the complete cluster. If you set this to `false`, it recovers +only the contents of the `.security` index, such as usernames and password +hashes, API keys, application privileges, role and role mapping definitions. +-- + +. Optionally, if you need to review and override the settings that were included +in the snapshot (by the `include_global_state` flag), cherry-pick and +<> that you +<> with the +`GET _cluster/settings` API. + +. If you pursue a complete point in time restore of the cluster, you also have +to restore configuration files. Again, this will restore non-security settings as +well. ++ +-- +This entails a straight-up filesystem copy of the backed up configuration files, +overwriting the contents of `$ES_PATH_CONF`, and restarting the node. This +needs to be done on *every node*. Depending on the extent of the differences +between your current cluster configuration and the restored configuration, you +may not be able to perform a rolling restart. If you are performing a full +restore of your configuration directory, we recommend a full cluster restart as +the safest option. Alternatively, you may wish to restore your configuration +files to a separate location on disk and use file comparison tools to review +the differences between your existing configuration and the restored +configuration. +-- diff --git a/docs/reference/administering/backup-cluster.asciidoc b/docs/reference/administering/backup-cluster.asciidoc index facf8402d39a0..5544af65bc172 100644 --- a/docs/reference/administering/backup-cluster.asciidoc +++ b/docs/reference/administering/backup-cluster.asciidoc @@ -18,6 +18,5 @@ To restore your cluster from a backup: include::backup-cluster-data.asciidoc[] include::backup-cluster-config.asciidoc[] -include::backup-security-config.asciidoc[] +include::backup-and-restore-security-config.asciidoc[] include::restore-cluster-data.asciidoc[] -include::restore-security-config.asciidoc[] \ No newline at end of file diff --git a/docs/reference/administering/restore-security-config.asciidoc b/docs/reference/administering/restore-security-config.asciidoc deleted file mode 100644 index 70ccb8530cf36..0000000000000 --- a/docs/reference/administering/restore-security-config.asciidoc +++ /dev/null @@ -1,114 +0,0 @@ -[role="xpack"] -[testenv="basic"] -[[restore-security-configuration]] -=== Restore a cluster's security configuration -++++ -Restore the security configuration -++++ - -NOTE: You can restore a snapshot of the `.security` index only if it was -created in a previous minor version in the same major version. The last minor -version of every major release can convert and read formats of the index for -both its major version and the next one. - -When you restore security configuration you have the option of doing a complete -restore of *all* configurations, including non-security ones, or to only restore -the contents of the `.security` index. As described in -<>, the second option comprises only -resource-type configurations. The first option has the advantage of restoring -a cluster to a clearly defined state from a past point in time. The second option -touches only security configuration resources, but it does not completely restore -the {security-features}. - -To restore your security configuration from a backup, first make sure that the -repository holding `.security` snapshots is installed: - -[source,js] --------------------------------------------------- -GET /_snapshot/my_backup --------------------------------------------------- -// CONSOLE - -[source,js] --------------------------------------------------- -GET /_snapshot/my_backup/snapshot_1 --------------------------------------------------- -// CONSOLE -// TEST[continued] - -Then log into one of the node hosts, navigate to {es} installation directory, -and follow these steps: - -. Add a new user with the `superuser` built-in role to the -{stack-ov}/file-realm.html[file realm]. -+ --- -For example, create a user named `restore_user`: -[source,shell] --------------------------------------------------- -bin/elasticsearch-users useradd restore_user -p password -r superuser --------------------------------------------------- --- - -. Using the previously created user, delete the existing `.security-6` or -`.security-7` index. -+ --- -[source,shell] --------------------------------------------------- -curl -u restore_user -X DELETE "localhost:9200/.security-*" --------------------------------------------------- -// NOTCONSOLE - -WARNING: After this step any authentication that relies on the `.security` -index will not work. This means that all API calls that authenticate with -native or reserved users will fail, as will any user that relies on a native role. -The file realm user we created in the step above will continue to work -because it is not stored in the `.security` index and uses the built-in -`superuser` role. - --- - -. Using the same user, restore the `.security` index from the snapshot. -+ --- -[source,shell] --------------------------------------------------- - curl -u restore_user -X POST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -H 'Content-Type: application/json' -d' - { - "indices": ".security-*", - "include_global_state": true <1> - } - ' --------------------------------------------------- -// NOTCONSOLE - -<1> The `include_global_state: true` is mandatory only for a complete restore. -This will restore the global cluster metadata, which contains configuration -information for the complete cluster. If you set this to `false`, it recovers -only the contents of the `.security` index, such as usernames and password -hashes, API keys, application privileges, role and role mapping definitions. --- - -. Optionally, if you need to review and override the settings that were included -in the snapshot (by the `include_global_state` flag), cherry-pick and -<> that you -<> with the -`GET _cluster/settings` API. - -. If you pursue a complete point in time restore of the cluster, you also have -to restore configuration files. Again, this will restore non-security settings as -well. -+ --- -This entails a straight-up filesystem copy of the backed up configuration files, -overwriting the contents of `$ES_PATH_CONF`, and restarting the node. This -needs to be done on *every node*. Depending on the extent of the differences -between your current cluster configuration and the restored configuration, you -may not be able to perform a rolling restart. If you are performing a full -restore of your configuration directory, we recommend a full cluster restart as -the safest option. Alternatively, you may wish to restore your configuration -files to a separate location on disk and use file comparison tools to review -the differences between your existing configuration and the restored -configuration. ---