Merge branch 'devel' into main

Author: Tristan Stevens

docs/freeipa.md

@@ -0,0 +1,135 @@
+# FreeIPA
+
+FreeIPA is a product that provides (not exhaustive):
+
+- CA
+- KDC
+- LDAP
+- Host management (including SSSD)
+
+The playbook is able to provision a FreeIPA server, or use an existing FreeIPA server, and automatically setup the cluster to make use of the CA, KDC and LDAP.
+
+Before continuing, a few questions you may want to ask yourself are:
+
+1. Are you using an existing FreeIPA server(s) or do you want the playbook to provision one?
+2. Do you want to use the FreeIPA CA to sign the certificates or do you want to sign them externally?
+3. Are you using AutoTLS or do you want the playbook to configure the host keys and certificates?
+
+## Common steps
+
+Regardless of how you choose to use FreeIPA in your deployment, you'll have to set the following variables via the definition file:
+
+- `krb5_realm` (e.g. `CLOUDERA.LOCAL`)
+- `krb5_kdc_admin_user` (e.g. the FreeIPA default `admin@{{ krb5_realm }}`)
+- `krb5_kdc_admin_password` (e.g. `{{ ipaadmin_password }}`)
+- `krb5_enc_types` (e.g. `aes256-cts aes128-cts`)
+
+You must also set `krb5_kdc_type: "Red Hat IPA"`.
+
+## Existing FreeIPA or playbook-provisioned?
+
+### Existing FreeIPA
+
+This case is simple:
+
+Please set `krb5_kdc_host` to you FreeIPA server hostname.
+
+### Playbook-provisioned
+
+Here, you'll need to add a host to `krb5_server` and set the following variables:
+
+- `ipaadmin_password`
+- `ipadm_password`
+
+The playbook will recognize that `krb5_kdc_type` is set to `Red Hat IPA` and bring up a FreeIPA server instead of MIT KDC.
+
+The playbook will not provision a firewall around the FreeIPA server.
+
+## FreeIPA CA signed certificates or externally signed certificates?
+
+In both cases, you'll want to refer to each CA certificate used (particularly important if you are using a different CA) by adding entries to `tls_ca_certs` e.g. (IPA CA)
+
+```
+tls_ca_certs:
+  - path: /etc/ipa/ca.crt
+    alias: ipaca
+```
+
+### FreeIPA CA signed certificates
+
+Here, nothing has to be done.
+
+Provided each host is enrolled as a FreeIPA client then the playbook will automatically sign (~~and enable the renewal of~~ ZOOKEEPER-3832) the host certificates using the hosts principal.
+
+### Externally signed certificates
+
+In this case, please set `skip_ipa_signing` to `true`.
+
+This will cause the playbook to stop after generating CSRs – identical to the non-FreeIPA case.
+
+## AutoTLS or playbook configured?
+
+### AutoTLS
+
+#### CM provisioned CA
+
+Remove any mention of TLS from the cluster definition and enable AutoTLS using the API or wizard.
+
+You may need to add the FreeIPA to the CA certs (via the API or wizard).
+
+#### FreeIPA or externally provisioned certificates
+
+Here, you'll want to unset any TLS configurations in the `definition.yml` file. This is because AutoTLS takes on the role of configuring the cluster here.
+
+You'll then need to enable AutoTLS using the certificates provisioned (by default) under `/opt/cloudera/security/pki` using the API https://blog.cloudera.com/auto-tls-in-cloudera-data-platform-data-center/ immediately after installing the Cloudera Manager.
+
+You can then continue with the playbook installation.
+
+## LDAP configs
+
+To setup LDAP in CM and services automatically, we'll need to first define an auth provider.
+
+For FreeIPA, it might look something like:
+
+```
+base_dn: "dc={{ (krb5_realm | lower).split('.') | join(',dc=') }}"
+user_dn: "cn=users,cn=accounts,{{ base_dn }}"
+group_dn: "cn=groups,cn=accounts,{{ base_dn }}"
+
+auth_providers:
+  freeipa:
+    ldap_bind_user_dn: "uid=admin,{{ user_dn }}"
+    ldap_bind_password: "{{ ipaadmin_password }}"
+    ldap_search_base:
+      user: "{{ user_dn }}"
+      group: "{{ group_dn }}"
+    ldap_object_class:
+      user: "person"
+      group: "groupofnames"
+    ldap_attribute:
+      user: "uid"
+      group: "cn"
+      member: "member"
+    type: LDAP
+    ldap_url: "ldaps://{{ groups.krb5_server | first }}"
+```
+
+Once the auth provider is defined, we need to configure the playbook to use it to:
+
+Configure CM:
+
+```
+cloudera_manager_external_auth:
+  provider: freeipa
+  external_first: yes
+  external_only: no
+  role_mappings:
+  - group: CMTestGroup1
+    roles: [ROLE_ADMIN]
+```
+
+Configure the services:
+
+```
+service_auth_provider: freeipa
+```

docs/getting-started.md

@@ -0,0 +1,7 @@
+# Getting Started
+
+cloudera.cluster is now an Ansible Galaxy collection and must therefore be used as part of a standalone playbook.
+
+For further details, please see https://github.com/cloudera-labs/cloudera-deploy
+
+Note: The docs in this section cover how to configure various aspects of this collection.

docs/how-to/clear-ca-server-state.md

@@ -0,0 +1,52 @@
+# How to: Clear CA Server state
+
+The CA Server (using `openssl ca`) can be used to sign the host certificates used to deploy the cluster.
+
+There may be cases where you wish to replace the host keys and certificates.
+
+# Replace the CA
+
+On each host in the cluster, move or delete the following directory:
+
+```
+/opt/cloudera/security/pki
+```
+
+On the `ca_server` host, move or delete the following directory:
+
+```
+/ca
+```
+
+Re-run the playbook.
+
+# Keep the CA but replace host certificates
+
+On each host in the cluster where you wish to replace the host keys and certificates, move or delete the following directory:
+
+```
+/opt/cloudera/security/pki
+```
+
+On the `ca_server` host, execute the following for each host you wish to replace:
+
+```
+openssl ca -config /ca/intermediate/openssl.cnf -revoke /ca/intermediate/cert/<host-certificate>.pem -passin pass:<ca-password>
+```
+
+__Note:__ The default CA password is `password` and can be configured by setting:
+
+- `ca_server_root_key_password`
+- `ca_server_intermediate_key_password`
+
+Now move or delete the corresponding host certificates from:
+
+```
+/ca/intermediate/cert
+```
+
+Re-run the playbook.
+
+__Note:__ You can skip the revoke step by setting `unique_subject` in `/ca/intermediate/index.txt.attr` to `no`.
+
+__Note:__ The CA Server is meant for testing only. Please use an enterprise CA Server in production.

docs/how-to/database-configuration.md

@@ -0,0 +1,80 @@
+# How to: Database Configuration
+
+This guide will run through the specifics of explicitly configuring databases for Cloudera Manager and the cluster services.
+
+It assumes you are already familiar with deploying standard clusters using these playbooks.
+
+### Create a cluster definition
+
+Create the cluster definition you wish to deploy.
+
+### Configure the Cloudera Manager database
+
+Add the following variables to `definition.yml`:
+
+- `cloudera_manager_database_host`
+- `cloudera_manager_database_name`
+- `cloudera_manager_database_user`
+- `cloudera_manager_database_password`
+
+You can also set the following variable if the Cloudera Manager database type is different from the global `database_type` setting:
+
+- `cloudera_manager_database_type`
+
+### Configure the cluster service databases
+
+**Note:** The type of every service database is expected match `database_type` (default postgres).
+
+For each cluster in the cluster list, add a dictionary entry `databases:`, level with `configs:`.
+
+Add an entry to the `databases:` dictionary for each service that requires a database. The service label should match the label in `services:` and `configs:`.
+
+For each service, set the following variables:
+
+- `name` (name of the database)
+- `host`
+- `user`
+- `password`
+
+Following these changes, your clusters should look like:
+
+```
+clusters:
+  - name: Test Cluster
+    services: [HDFS, ...]
+    databases:
+      HIVE:
+        name: "metastore"
+        host: "db-1.example.com"
+        user: "hive"
+        password: "{{ hive_password }}"
+      ...
+    ...
+  ...
+```
+
+### Configure the mgmt databases
+
+**Note:** The type of every mgmt role database is expected match `database_type` (default postgres).
+
+To config mgmt databases, following the method explained above for clusters, placing `databases:` in `mgmt:`.
+
+Following these changes, mgmt should look like:
+
+```
+mgmt:
+  name: Cloudera Management Service
+  services: [REPORTSMANAGER, ...]
+  databases:
+    REPORTSMANAGER:
+      name: "rman"
+      host: "db-2.example.com"
+      user: "rman"
+      password: "{{ rman_password }}"
+    ...
+  ...
+```
+
+### **Deploy!**
+
+**Note:** If the configured database host is provisioned by the playbook (part of the `db_servers` host group) the database will be created automatically.

docs/how-to/externally-signed.md

@@ -0,0 +1,98 @@
+# How to: Setup externally signed TLS certificates
+
+The playbook is able to provision an internal CA and automatically sign the host certificates.
+
+However, it is more likely that an external CA will be used in production environments.
+
+This guide will run through specifics for deploying clusters using an external CA to sign the host certificates.
+
+At a high-level, the playbook generates and distributes the TLS keys and certificates in three steps:
+
+- Phase 1 (generation)
+  * Create the host keys
+  * Generate the keystores
+  * Generate the CSRs
+  * Copy the CSRs to the Ansible controller
+- Phase 2 (signing)
+  * Sign the certificates
+- Phase 3 (installation)
+  * Copy the signed certificates to the hosts
+  * Copy the CA certificates to the hosts
+
+When `ca_server` is used, this all occurs automatically.
+
+However, when an external CA is used, a few manual steps are required to complete the deployment:
+
+Starting with the definition, assuming nothing has been deployed:
+
+## Ensure that `ca_server` is **not** present in the inventory
+
+The `ca_server` is the playbook's internal CA.
+
+As we wish to use an external CA, it should be omitted from the inventory file.
+
+## Set `tls_ca_certs` to point to the external CA certificates
+
+Here, we need to set `tls_ca_certs` in `defintion.yml` to point to the external CA certificates (on the Ansible controller):
+
+```
+tls_ca_certs:
+  - alias: ipaca
+    path: /root/ca.crt
+```
+
+In this example, our CA certificate exists on the Ansible controller at `/root/ca.crt`.
+
+**Note:** If you have an intermediate CA and a root CA, please include both certificates here.
+
+## Run the playbook deployment as usual
+
+If everything is configured correctly, it should run as usual until we reach `prepare_tls.yml`.
+
+Here, in `prepare_tls.yml`, it will fail with a message:
+
+> Signed cert for <hostname> could not be found. If manual signing is required, do this now and re-run the playbook with 'tls_signed_certs_dir' variable set.
+
+E.g.
+
+```
+TASK [security/tls_install_certs : fail] **************************************************************************************************************
+fatal: [host-1.example.com] FAILED! => {"changed": false, "msg": "\"Signed cert for host-1.example.com could not be found. If manual signing is required, do this now and re-run the playbook with 'tls_signed_certs_dir' variable set.\n"}
+fatal: [host-2.example.com] FAILED! => {"changed": false, "msg": "\"Signed cert for host-1.example.com could not be found. If manual signing is required, do this now and re-run the playbook with 'tls_signed_certs_dir' variable set.\n"}
+fatal: [host-3.example.com] FAILED! => {"changed": false, "msg": "\"Signed cert for host-1.example.com could not be found. If manual signing is required, do this now and re-run the playbook with 'tls_signed_certs_dir' variable set.\n"}
+fatal: [host-4.example.com] FAILED! => {"changed": false, "msg": "\"Signed cert for host-1.example.com could not be found. If manual signing is required, do this now and re-run the playbook with 'tls_signed_certs_dir' variable set.\n"}
+```
+
+This is expected.
+
+At this point, the playbook will have generated the TLS keys and CSRs and copied the CSRs to the Ansible controller (stage 1).
+
+It is up to us now to sign the CSRs and copy the signed certificates back to the Ansible controller:
+
+## Sign the CSRs generated by the playbook
+
+You will find copies of the TLS CSRs in `{{ local_temp_dir }}/csrs` on the Ansible controller (default `/tmp/csrs`).
+
+```
+# ls /tmp/csrs
+host-1.example.com.csr  host-3.example.com.csr
+host-2.example.com.csr  host-4.example.com.csr
+```
+
+Sign the CSRs and copy the signed certificates to the Ansible controller using the same filename, replacing `.csr` with `.pem`.
+
+Place these signed certificates in `{{ local_temp_dir }}/certs` on the Ansible controller (default `/tmp/certs`).
+
+```
+# ls /tmp/certs
+host-1.example.com.pem  host-3.example.com.pem
+host-2.example.com.pem  host-4.example.com.pem
+```
+
+## Rerun the playbook
+
+The playbook can be restarted now that the signed certificates exist on the Ansible controller (stage 2).
+
+The playbook will distribute the new signed certificates and continue as usual.
+
+No other steps are required.