[DOC] Backup & Restore Security Configuration

docs/reference/administering.asciidoc

@@ -9,17 +9,4 @@ cluster.
 
 --
 
-[[backup-cluster]]
-== Back up a cluster
-
-As with any software that stores data, it is important to routinely back up your
-data. {es} replicas provide high availability during runtime; they enable you to
-tolerate sporadic node loss without an interruption of service.
-
-Replicas do not provide protection from catastrophic failure, however. For that,
-you need a real backup of your cluster—a complete copy in case something goes
-wrong.
-
-To back up your cluster, you can use the <<modules-snapshots,snapshot API>>.
-
-include::{es-repo-dir}/modules/snapshots.asciidoc[tag=snapshot-intro]
+include::administering/backup-cluster.asciidoc[]

docs/reference/administering/backup-and-restore-security-config.asciidoc

@@ -0,0 +1,281 @@
+[role="xpack"]
+[testenv="basic"]
+[[security-backup]]
+=== Back up a cluster's security configuration
+++++
+<titleabbrev>Back up the security configuration</titleabbrev>
+++++
+
+Security configuration information resides in two places:
+<<backup-security-file-based-configuration,files>> and
+<<backup-security-index-configuration,indices>>.
+
+[discrete]
+[[backup-security-file-based-configuration]]
+==== Back up file-based security configuration
+
+{es} {security-features} are configured using the <<security-settings,
+`xpack.security` namespace>> inside the `elasticsearch.yml` and
+`elasticsearch.keystore` files. In addition there are several other
+<<security-files, extra configuration files>> inside the same `ES_PATH_CONF`
+directory. These files define roles and role mappings and
+<<configuring-file-realm, configure the file realm>>. Some of the
+settings specify file paths to security-sensitive data, such as TLS keys and
+certificates for the HTTP client and inter-node communication and private key files for
+<<ref-saml-settings, SAML>>, <<ref-oidc-settings, OIDC>> and the
+<<ref-kerberos-settings, Kerberos>> realms. All these are also stored inside
+`ES_PATH_CONF`; the path settings are relative.
+
+IMPORTANT: The `elasticsearch.keystore`, TLS keys and SAML, OIDC, and Kerberos
+realms private key files require confidentiality. This is crucial when files
+are copied to the backup location, as this increases the surface for malicious
+snooping.
+
+To back up all this configuration you can use a <<backup-cluster-configuration,
+conventional file-based backup>>, as described in the previous section.
+
+[NOTE]
+====
+
+* File backups must run on every cluster node.
+* File backups will store non-security configuration as well. Backing-up
+only {security-features} configuration is not supported. A backup is a
+point in time record of state of the complete configuration.
+
+====
+
+[discrete]
+[[backup-security-index-configuration]]
+==== Back up index-based security configuration
+
+{es} {security-features} store system configuration data inside a
+dedicated index. This index is named `.security-6` in the {es} 6.x versions and
+`.security-7` in the 7.x releases. The `.security` alias always points to the
+appropriate index. This index contains the data which is not available in
+configuration files and *cannot* be reliably backed up using standard
+filesystem tools. This data describes:
+
+* the definition of users in the native realm (including hashed passwords)
+* role definitions (defined via the <<security-api-put-role,create roles API>>)
+* role mappings (defined via the
+  <<security-api-put-role-mapping,create role mappings API>>)
+* application privileges
+* API keys
+
+The `.security` index thus contains resources and definitions in addition to
+configuration information. All of that information is required in a complete
+{security-features} backup.
+
+Use the <<modules-snapshots, standard {es} snapshot functionality>> to backup
+`.security`, as you would for any <<backup-cluster-data, other data index>>.
+For convenience, here are the complete steps:
+
+. Create a repository that you can use to backup the `.security` index.
+It is preferable to have a <<backup-security-repos, dedicated repository>> for
+this special index. If you wish, you can also snapshot the system indices for other {stack} components to this repository. 
++
+--
+[source,js]
+-----------------------------------
+PUT /_snapshot/my_backup
+{
+  "type": "fs",
+  "settings": {
+    "location": "my_backup_location"
+  }
+}
+-----------------------------------
+// CONSOLE
+
+The user calling this API must have the elevated `manage` cluster privilege to
+prevent non-administrators exfiltrating data.
+
+--
+
+. Create a user and assign it only the built-in `snapshot_user` role.
++
+--
+The following example creates a new user `snapshot_user` in the
+{stack-ov}/native-realm.html[native realm], but it is not important which
+realm the user is a member of:
+
+[source,js]
+--------------------------------------------------
+POST /_security/user/snapshot_user
+{
+  "password" : "secret",
+  "roles" : [ "snapshot_user" ]
+}
+--------------------------------------------------
+// CONSOLE
+// TEST[skip:security is not enabled in this fixture]
+
+--
+
+. Create incremental snapshots authorized as `snapshot_user`.
++
+--
+The following example shows how to use the create snapshot API to backup
+the `.security` index to the `my_backup` repository:
+
+[source,js]
+--------------------------------------------------
+PUT /_snapshot/my_backup/snapshot_1
+{
+  "indices": ".security",
+  "include_global_state": true <1>
+}
+--------------------------------------------------
+// CONSOLE
+// TEST[continued]
+
+<1> This parameter value captures all the persistent settings stored in the
+global cluster metadata as well as other configurations such as aliases and
+stored scripts. Note that this includes non-security configuration and that it complements but does not replace the
+<<backup-cluster-configuration, filesystem configuration files backup>>.
+
+--
+
+IMPORTANT: The index format is only compatible within a single major version,
+and cannot be restored onto a version earlier than the version from which it
+originated. For example, you can restore a security snapshot from 6.6.0 into a
+6.7.0 cluster, but you cannot restore it to a cluster running {es} 6.5.0 or 7.0.0.
+
+[discrete]
+[[backup-security-repos]]
+===== Controlling access to the backup repository
+
+The snapshot of the security index will typically contain sensitive data such
+as user names and password hashes. Because passwords are stored using
+<<hashing-settings, cryptographic hashes>>, the disclosure of a snapshot would
+not automatically enable a third party to authenticate as one of your users or
+use API keys. However, it would disclose confidential information.
+
+It is also important that you protect the integrity of these backups in case
+you ever need to restore them. If a third party is able to modify the stored
+backups, they may be able to install a back door that would grant access if the
+snapshot is loaded into an {es} cluster.
+
+We recommend that you:
+
+* Snapshot the `.security` index in a dedicated repository, where read and write
+access is strictly restricted and audited.
+* If there are indications that the snapshot has been read, change the passwords
+of the users in the native realm and revoke API keys.
+* If there are indications that the snapshot has been tampered with, do not
+restore it. There is currently no option for the restore process to detect
+malicious tampering.
+
+[[restore-security-configuration]]
+=== Restore a cluster's security configuration
+++++
+<titleabbrev>Restore the security configuration</titleabbrev>
+++++
+
+NOTE: You can restore a snapshot of the `.security` index only if it was
+created in a previous minor version in the same major version. The last minor
+version of every major release can convert and read formats of the index for
+both its major version and the next one.
+
+When you restore security configuration you have the option of doing a complete
+restore of *all* configurations, including non-security ones, or to only restore
+the contents of the `.security` index. As described in
+<<backup-security-index-configuration>>, the second option comprises only
+resource-type configurations. The first option has the advantage of restoring
+a cluster to a clearly defined state from a past point in time. The second option
+touches only security configuration resources, but it does not completely restore
+the {security-features}.
+
+To restore your security configuration from a backup, first make sure that the
+repository holding `.security` snapshots is installed:
+
+[source,js]
+--------------------------------------------------
+GET /_snapshot/my_backup
+--------------------------------------------------
+// CONSOLE
+// TEST[continued]
+
+[source,js]
+--------------------------------------------------
+GET /_snapshot/my_backup/snapshot_1
+--------------------------------------------------
+// CONSOLE
+// TEST[continued]
+
+Then log into one of the node hosts, navigate to {es} installation directory,
+and follow these steps:
+
+. Add a new user with the `superuser` built-in role to the
+{stack-ov}/file-realm.html[file realm].
++
+--
+For example, create a user named `restore_user`:
+[source,shell]
+--------------------------------------------------
+bin/elasticsearch-users useradd restore_user -p password -r superuser
+--------------------------------------------------
+--
+
+. Using the previously created user, delete the existing `.security-6` or
+`.security-7` index.
++
+--
+[source,shell]
+--------------------------------------------------
+curl -u restore_user -X DELETE "localhost:9200/.security-*"
+--------------------------------------------------
+// NOTCONSOLE
+
+WARNING: After this step any authentication that relies on the `.security`
+index will not work. This means that all API calls that authenticate with
+native or reserved users will fail, as will any user that relies on a native role.
+The file realm user we created in the step above will continue to work
+because it is not stored in the `.security` index and uses the built-in
+`superuser` role.
+
+--
+
+. Using the same user, restore the `.security` index from the snapshot.
++
+--
+[source,shell]
+--------------------------------------------------
+ curl -u restore_user -X POST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -H 'Content-Type: application/json' -d'
+ {
+    "indices": ".security-*",
+    "include_global_state": true <1>
+ }
+ '
+--------------------------------------------------
+// NOTCONSOLE
+
+<1> The `include_global_state: true` is mandatory only for a complete restore.
+This will restore the global cluster metadata, which contains configuration
+information for the complete cluster. If you set this to `false`, it recovers
+only the contents of the `.security` index, such as usernames and password
+hashes, API keys, application privileges, role and role mapping definitions.
+--
+
+. Optionally, if you need to review and override the settings that were included
+in the snapshot (by the `include_global_state` flag), cherry-pick and
+<<cluster-update-settings,apply the persistent settings>> that you
+<<backup-cluster-configuration, have extracted>> with the
+`GET _cluster/settings` API.
+
+. If you pursue a complete point in time restore of the cluster, you also have
+to restore configuration files. Again, this will restore non-security settings as
+well.
++
+--
+This entails a straight-up filesystem copy of the backed up configuration files,
+overwriting the contents of `$ES_PATH_CONF`, and restarting the node. This
+needs to be done on *every node*. Depending on the extent of the differences
+between your current cluster configuration and the restored configuration, you
+may not be able to perform a rolling restart. If you are performing a full
+restore of your configuration directory, we recommend a full cluster restart as
+the safest option. Alternatively, you may wish to restore your configuration
+files to a separate location on disk and use file comparison tools to review
+the differences between your existing configuration and the restored
+configuration.
+--

docs/reference/administering/backup-cluster-config.asciidoc

@@ -0,0 +1,62 @@
+[[backup-cluster-configuration]]
+=== Back up a cluster's configuration
+++++
+<titleabbrev>Back up the cluster configuration</titleabbrev>
+++++
+
+In addition to backing up the data in a cluster, it is important to back up its configuration--especially when the cluster becomes large and difficult to
+reconstruct.
+
+Configuration information resides in
+<<config-files-location, regular text files>> on every cluster node. Sensitive
+setting values such as passwords for the {watcher} notification servers, are
+specified inside a binary secure container, the
+<<secure-settings, elasticsearch.keystore>> file. Some setting values are
+file paths to the associated configuration data, such as the ingest geo ip
+database. All these files are contained inside the `ES_PATH_CONF` directory.
+
+NOTE: All changes to configuration files are done by manually editing the files
+or using command line utilities, but *not* through APIs. In practice, these
+changes are infrequent after the initial setup.
+
+We recommend that you take regular (ideally, daily) backups of your {es} config
+(`$ES_PATH_CONF`) directory using the file backup software of your choice.
+
+TIP: We recommend that you have a configuration management plan for these
+configuration files. You may wish to check them into version control, or
+provision them though your choice of configuration management tool.
+
+Some of these files may contain sensitive data such as passwords and TLS keys,
+therefore you should investigate whether your backup software and/or storage
+solution are able to encrypt this data.
+
+Some settings in configuration files might be overridden by
+<<cluster-update-settings,cluster settings>>. You can capture these settings in
+a *data* backup snapshot by specifying the `include_global_state: true` (default)
+parameter for the snapshot API. Alternatively, you can extract these
+configuration values in text format by using the
+<<cluster-get-settings, get settings API>>:
+
+[source,js]
+--------------------------------------------------
+GET _cluster/settings?pretty&flat_settings&filter_path=persistent
+--------------------------------------------------
+//CONSOLE
+//TEST
+
+You can store the output of this as a file together with the rest of
+configuration files.
+
+[NOTE]
+====
+
+* Transient settings are not considered for backup.
+* {es} {security-features} store configuration data such as role definitions and
+API keys inside a dedicate special index. This "system" data,
+complements the <<secure-settings, security settings>> configuration and should
+be <<backup-security-index-configuration, backed up as well>>.
+* Other {stack} components, like Kibana and {ml-cap}, store their configuration
+data inside other dedicated indices. From the {es} perspective these are just data
+so you can use the regular <<backup-cluster-data, data backup>> process.
+
+====