diff --git a/private-cloud/base/aws-iaas/.gitignore b/private-cloud/base/aws-iaas/.gitignore new file mode 100644 index 0000000..e6cda98 --- /dev/null +++ b/private-cloud/base/aws-iaas/.gitignore @@ -0,0 +1,45 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Ansible Navigator assets +ansible-navigator.log +runs +context + +# Terraform deployments +tf_deployment* + +# Local .terraform directories +**/.terraform/* + +# .tfstate files +*.tfstate +*.tfstate.* + +# .lock files +*.terraform.lock.hcl + +# .tfvars files +*.tfvars + +# Static inventory files +inventory_static* + +# Per-user configuration files +config.yml + +# Per-deployment artifacts +*-DEPLOYMENT.html +*-DEPLOYMENT.md +*.ca.crt diff --git a/private-cloud/base/aws-iaas/README.md b/private-cloud/base/aws-iaas/README.md new file mode 100644 index 0000000..224c565 --- /dev/null +++ b/private-cloud/base/aws-iaas/README.md @@ -0,0 +1,182 @@ +# PvC Base Cluster on AWS IaaS + +> Constructs a CDP Private Cloud Base cluster running on AWS IaaS. + +## Known Issues + +| Issue | Description | Workaround | +|-------|-------------|------------| +| Cluster instances unavailable after the `external_setup.yml` Playbook | The cluster EC2 instances become unavailable after the `external_setup.yml` Playbook. During subsequent playbooks the hosts becomes unreachable and in the EC2 console the VM instances fail the reachability health check. | Restart the EC2 instances via the console. | + +## Requirements + +To run, you need: + +* Docker (or a Docker alternative) +* `ansible-navigator` +* AWS credentials +* CDP Private Cloud Base license file +* SSH key(s) for bastion/jump host and cluster + +### Configuration Variables + +Configuration is passed via environment variables and an user-managed configuration file. + +#### Environment Variables + +* Set up the following definition environment variables: + + | Variable | Description | Status | + |----------|-------------|--------| + | `SSH_PUBLIC_KEY_FILE` | File path to the SSH public key that will be uploaded to the cloud provider (using the `name_prefix` variable as the key label). E.g. `/Users/example/.ssh/demo_ops.pub` | Mandatory | + | `SSH_PRIVATE_KEY_FILE` | File path to the SSH private key. E.g. `/Users/example/.ssh/demo_ops` | Mandatory | + | `CDP_LICENSE_FILE` | File path to a CDP Private Cloud Base license. E.g. `/Users/example/Documents/example_cloudera_license.txt` | Mandatory | + | `AWS_PROFILE` | The profile label for your AWS credentials. Otherwise, use the associated `AWS_*` parameters. | Mandatory | + +#### Configuration file variables + +Copy `config-template.yml` to `config.yml` and edit this user-facing configuration file to match your particular deployment. + +> [!IMPORTANT] +> `name_prefix` should be 4-7 characters and is the "primary key" for the deployment. + +```yaml +name_prefix: "{{ mandatory }}" # Unique identifier for the deployment +infra_region: "us-east-2" +domain: "{{ name_prefix }}.cldr.example" # The deployment subdomain +realm: "CLDR.DEPLOYMENT" # The Kerberos realm +common_password: "Example776" # For external services +admin_password: "Example776" # For Cloudera-related services +deployment_tags: + deployment: "{{ name_prefix }}" + deploy-tool: cloudera-deploy +``` + +## Execution + +## All-in-One + +You can run all of the following steps at once, if you wish: + +```bash +ansible-navigator run \ + pre_setup.yml \ + external_setup.yml \ + internal_setup.yml \ + base_setup.yml \ + summary.yml \ + -e @definition.yml \ + -e @config.yml +``` + +### Pre-setup Playbook + +This definition-specific playbook includes tasks such as: + +* Instructure provisioning +* FreeIPA DNS and KRB services provisioning + +Run the following command + +```bash +ansible-navigator run pre_setup.yml \ + -e @definition.yml \ + -e @config.yml +``` + +Once the pre-setup playbook completes confirm that: + +* You can connect to each node via the inventory - see [Confirm SSH Connectivity](#confirm-ssh-connectivity) for help. You can also run `ansible-navigator run validate_dns_lookups.yml` to check connectivity and DNS. +* Connect to FreeIPA UI and login with the `IPA_USER` and `IPA_PASSWORD` credentials in the configuration file. See [Cluster Access](#cluster-access) for details. + +### Platform Playbooks + +These playbooks configure and deploy PVC Base. They use the infrastructure provisioned. + +Tasks include: + +* System/host configuration +* Cloudera Manager server and agent installation and configuration +* Cluster template imports + +Run the following: + +```bash +# Run the 'external' system configuration +ansible-navigator run external_setup.yml \ + -e @definition.yml \ + -e @config.yml +``` + +```bash +# Run the 'internal' Cloudera installations and configurations +ansible-navigator run internal_setup.yml \ + -e @definition.yml \ + -e @config.yml +``` + +```bash +# Run the Cloudera cluster configuration and imports +ansible-navigator run base_setup.yml \ + -e @definition.yml \ + -e @config.yml +``` + +```bash +# Produce a deployment summary and retrieve the FreeIPA CA certificate +ansible-navigator run summary.yml \ + -e @definition.yml \ + -e @config.yml +``` + +## Cluster Access + +Once the cluster is up, you can access all of the UIs within, including the FreeIPA sidecar, via a SSH tunnel: + +```bash +ssh -D 8157 -q -C -N ec2-user@ +``` + +Use a SOCKS5 proxy switcher in your browser (an example is the SwitchyOmega browser extension). + +In the SOCKS5 proxy configuration, set _Protocol_ to `SOCKS5`, _Server_ to `localhost`, and _Port_ to `8157`. Ensure the SOCKS5 proxy is active when clicking on the CDP UI that you wish to access. + +> [!CAUTION] +> You will get a SSL warning for the self-signed certificate; this is expected given this particular definition as the local FreeIPA server has no upstream certificates. However, you can install this CA certificate to remove this notification. + +In addition, you can log into the jump host via SSH and get to any of the servers within the cluster. Remember to forward your SSH key! + +```bash +ssh -A -C -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ec2-user@ +``` + +> [!NOTE] +> The above assume you are using the default AMI image set in the Terraform configuration. If not, adjust the SSH user appropriately. + +## Teardown + +Run the following: + +```bash +ansible-navigator run pre_teardown.yml \ + -e @definition.yml \ + -e @config.yml +``` + +You can also run the direct Terraform command: + +```bash +ansible-navigator exec -- terraform -chdir=tf_proxied_cluster destroy -auto-approve +``` + +## Troubleshooting + +### Confirm SSH Connectivity + +Run the following: + +```bash +ansible-navigator exec -- ansible -m ansible.builtin.ping -i inventory.yml all +``` + +This will check to see if the inventory file is well constructed and the hosts are available via SSH. diff --git a/private-cloud/base/aws-iaas/ansible-navigator.yml b/private-cloud/base/aws-iaas/ansible-navigator.yml new file mode 100644 index 0000000..400a074 --- /dev/null +++ b/private-cloud/base/aws-iaas/ansible-navigator.yml @@ -0,0 +1,70 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +--- + +ansible-navigator: + playbook-artifact: + save-as: "runs/{playbook_name}-{time_stamp}.json" + + ansible-runner: + artifact-dir: runs + rotate-artifacts-count: 3 + + logging: + level: debug + append: False + + ansible: + inventory: + entries: + - inventory.yml + + execution-environment: + container-engine: docker + enabled: True + environment-variables: + pass: + - AWS_PROFILE + - SSH_PUBLIC_KEY_FILE + - SSH_PRIVATE_KEY_FILE + - CDP_LICENSE_FILE + #- IPA_USER + #- IPA_PASSWORD + set: + ANSIBLE_SSH_CONTROL_PATH: "/dev/shm/cp%%h-%%p-%%r" + ANSIBLE_CALLBACK_WHITELIST: "ansible.posix.profile_tasks" + ANSIBLE_GATHERING: "smart" + ANSIBLE_DEPRECATION_WARNINGS: False + ANSIBLE_HOST_KEY_CHECKING: False + ANSIBLE_SSH_RETRIES: 10 + image: ghcr.io/cloudera-labs/cldr-runner:aws-latest + pull: + arguments: + - "--tls-verify=false" + volume-mounts: + - src: "${SSH_PUBLIC_KEY_FILE}" + dest: "${SSH_PUBLIC_KEY_FILE}" + - src: "${SSH_PRIVATE_KEY_FILE}" + dest: "${SSH_PRIVATE_KEY_FILE}" + - src: "${CDP_LICENSE_FILE}" + dest: "${CDP_LICENSE_FILE}" + - src: "~/.aws" + dest: "/runner/.aws" + options: "Z" + - src: "~/.ssh" + dest: "/runner/.ssh" + options: "Z" + container-options: + - "--network=host" diff --git a/private-cloud/base/aws-iaas/base_postfix.yml b/private-cloud/base/aws-iaas/base_postfix.yml new file mode 100644 index 0000000..b047714 --- /dev/null +++ b/private-cloud/base/aws-iaas/base_postfix.yml @@ -0,0 +1,52 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +--- + +- name: Postfix CDP Private Cloud clusters + hosts: localhost + connection: local + gather_facts: yes + vars: + definition_path: "./" + tasks: + - name: Set of deployment variables from definition.yml + ansible.builtin.include_role: + name: cloudera.exe.init_deployment + public: yes + when: init__completed is undefined + + - name: Prepare inventory for PvC Plays + ansible.builtin.include_role: + name: cloudera.exe.init_deployment + public: yes + tasks_from: prep_pvc.yml + tags: + - always + +- name: Init run tasks for all nodes + hosts: all + gather_facts: no + tasks: + - name: Group hosts by host template and TLS + ansible.builtin.include_role: + name: cloudera.cluster.deployment.groupby + + - name: Check connectivity to Inventory + ansible.builtin.wait_for_connection: + tags: + - always + +- name: Postfix clusters for CDP Private Cloud + ansible.builtin.import_playbook: cloudera.exe.pvc_base_postfix.yml diff --git a/private-cloud/base/aws-iaas/base_setup.yml b/private-cloud/base/aws-iaas/base_setup.yml new file mode 100644 index 0000000..a3bda48 --- /dev/null +++ b/private-cloud/base/aws-iaas/base_setup.yml @@ -0,0 +1,52 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +--- + +- name: Set up CDP Private Cloud clusters + hosts: localhost + connection: local + gather_facts: yes + vars: + definition_path: "./" + tasks: + - name: Set of deployment variables from definition.yml + ansible.builtin.include_role: + name: cloudera.exe.init_deployment + public: yes + when: init__completed is undefined + + - name: Prepare inventory for PvC Plays + ansible.builtin.include_role: + name: cloudera.exe.init_deployment + public: yes + tasks_from: prep_pvc.yml + tags: + - always + +- name: Init run tasks for all nodes + hosts: all + gather_facts: no + tasks: + - name: Group hosts by host template and TLS + ansible.builtin.include_role: + name: cloudera.cluster.deployment.groupby + + - name: Check connectivity to Inventory + ansible.builtin.wait_for_connection: + tags: + - always + +- name: Set up clusters for CDP Private Cloud + ansible.builtin.import_playbook: cloudera.exe.pvc_base_setup.yml diff --git a/private-cloud/base/aws-iaas/cluster.yml b/private-cloud/base/aws-iaas/cluster.yml new file mode 100644 index 0000000..70322cb --- /dev/null +++ b/private-cloud/base/aws-iaas/cluster.yml @@ -0,0 +1,247 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +--- + +clusters: + - name: PVC-Base + type: base + services: [ATLAS, HBASE, HDFS, HIVE, HIVE_ON_TEZ, INFRA_SOLR, KAFKA, OZONE, RANGER, SPARK_ON_YARN, TEZ, YARN, ZOOKEEPER] + security: + kerberos: true + tls: true + repositories: "{{ repositories }}" + configs: + ATLAS: + SERVICEWIDE: + kerberos.auth.enable: true + ATLAS_SERVER: + atlas_authentication_method_file: true + atlas_admin_password: "{{ common_password }}" + atlas_sso_knox_enabled: false + atlas_authentication_method_trustedproxy: false + atlas_authentication_method_pam: false + atlas_authentication_method_ldap: true + atlas_authentication_method_ldap_url: "{{ auth_provider.ldap_url }}" + atlas_authentication_method_ldap_bind_dn: "{{ auth_provider.ldap_bind_user_dn }}" + atlas_authentication_method_ldap_bind_password: "{{ auth_provider.ldap_bind_password }}" + atlas_authentication_method_ldap_userDNpattern: "uid={0},{{ auth_provider.ldap_search_base.user }}" + atlas_authentication_method_ldap_base_dn: "{{ auth_provider.ldap_search_base.user }}" + atlas_authentication_method_ldap_user_searchfilter: "{{ auth_provider.ldap_search_filter.user }}" + atlas_authentication_method_ldap_groupSearchBase: "{{ auth_provider.ldap_search_base.group }}" + atlas_authentication_method_ldap_groupSearchFilter: "{{ auth_provider.ldap_search_filter.member }}" + atlas_authentication_method_ldap_groupRoleAttribute: "{{ auth_provider.ldap_attr_group_name }}" + atlas_authentication_method_ldap_type: "{{ auth_provider.type | lower }}" + atlas_authentication_method_ldap_default_role: "ROLE_ADMIN" + HBASE: + SERVICEWIDE: + hadoop_secure_web_ui: true + hbase_graceful_stop_timeout: 5 + REGIONSERVER: + hbase_regionserver_java_heapsize: 268435456 + MASTER: + hbase_master_java_heapsize: 134217728 + GATEWAY: + hbase_client_java_heapsize: 134217728 + # NOTE: Have removed /mnt from the various HFDS directories + HDFS: + SERVICEWIDE: + hdfs_verify_ec_with_topology_enabled: false + enable_ranger_authorization: true + hadoop_secure_web_ui: false + trusted_realms: "{{ realm | upper }}, {{ realm | lower }}" + DATANODE: + datanode_java_heapsize: 134217728 + dfs_data_dir_list: /dfs/dn + dfs_datanode_max_locked_memory: 0 + dfs_datanode_failed_volumes_tolerated: 0 + GATEWAY: + hdfs_client_java_heapsize: 134217728 + NAMENODE: + dfs_name_dir_list: /dfs/nn + namenode_java_heapsize: 134217728 + role_config_suppression_namenode_java_heapsize_minimum_validator: true + SECONDARYNAMENODE: + secondary_namenode_java_heapsize: 134217728 + fs_checkpoint_dir_list: /dfs/snn + HIVE: + SERVICEWIDE: + ranger_plugin_urlauth_filesystem_schemes: "file:,wasb:,adl:" + GATEWAY: + hive_client_java_heapsize: 134217728 + HIVEMETASTORE: + hive_metastore_java_heapsize: 268435456 + metastore_canary_health_enabled: false + HIVE_ON_TEZ: + SERVICEWIDE: + ranger_plugin_urlauth_filesystem_schemes: "file:,wasb:,adl:" + KAFKA: + SERVICEWIDE: + kerberos.auth.enable: true + offsets.topic.replication.factor: 1 + transaction.state.log.replication.factor: 1 + transaction.state.log.min.isr: 1 + producer.metrics.enable: false + service_config_suppression_kafka_broker_count_validator: true + service_config_suppression_offsets.topic.replication.factor: true + service_config_suppression_transaction.state.log.replication.factor: true + KAFKA_BROKER: + graceful_stop_timeout: 5 + OZONE: + SERVICEWIDE: + ozone.security.enabled: true + ozone.service.id: ozone1 + ozone.security.http.kerberos.enabled: false + service_config_suppression_ozone_manager_count_validator: true + service_config_suppression_storage_container_manager_count_validator: true + OZONE_MANAGER: + om_max_heap_size: 2048 + STORAGE_CONTAINER_MANAGER: + scm_max_heap_size: 2048 + OZONE_DATANODE: + ozone-conf/ozone-site.xml_role_safety_valve: | + + hdds.datanode.client.port + 9874 + + ozone_datanode_heap_size: 2048 + OZONE_PROMETHEUS: + ozone.prometheus.http-port: 19090 + RANGER: + SERVICEWIDE: + keyadmin_user_password: "{{ common_password }}" + rangeradmin_user_password: "{{ common_password }}" + rangertagsync_user_password: "{{ common_password }}" + rangerusersync_user_password: "{{ common_password }}" + RANGER_ADMIN: + ranger_admin_max_heap_size: 1024 + ranger_authentication_method: "{{ auth_provider.type | upper }}" + ranger.ldap.url: "{{ auth_provider.ldap_url }}" + ranger.ldap.bind.dn: "{{ auth_provider.ldap_bind_user_dn }}" + ranger_ldap_bind_password: "{{ auth_provider.ldap_bind_password }}" + ranger.ldap.user.dnpattern: "uid={0},{{ auth_provider.ldap_search_base.user }}" + ranger.ldap.base.dn: "{{ auth_provider.ldap_base_dn }}" + ranger.ldap.user.searchfilter: "{{ auth_provider.ldap_search_filter.user }}" + ranger.ldap.group.searchfilter: "{{ auth_provider.ldap_search_filter.member }}" + ranger.ldap.group.searchbase: "{{ auth_provider.ldap_search_base.group }}" + ranger.ldap.group.roleattribute: "{{ auth_provider.ldap_attr_group_name }}" + ranger.ldap.referral: "follow" + RANGER_TAGSYNC: + ranger_tagsync_max_heap_size: 1024 + RANGER_USERSYNC: + ranger_usersync_max_heap_size: 1024 + ranger.usersync.ldap.binddn: "{{ auth_provider.ldap_bind_user_dn }}" + ranger.usersync.ldap.url: "{{ auth_provider.ldap_url }}" + ranger_usersync_ldap_ldapbindpassword: "{{ auth_provider.ldap_bind_password }}" + ranger.usersync.ldap.referral: follow + ranger.usersync.ldap.username.caseconversion: lower + ranger.usersync.ldap.groupname.caseconversion: lower + ranger.usersync.ldap.grouphierarchylevels: 0 + # ranger.usersync.group.based.role.assignment.rules: '&ROLE_SYS_ADMIN:g:field-se' + ranger.usersync.group.memberattributename: "{{ auth_provider.ldap_attr_group_membership }}" + ranger.usersync.group.nameattribute: "{{ auth_provider.ldap_attr_group_name }}" + ranger.usersync.group.objectclass: "{{ auth_provider.ldap_object_class.group }}" # group + ranger.usersync.group.searchbase: "{{ auth_provider.ldap_search_base.group }}" +# ranger.usersync.group.searchfilter: "" +# ranger.usersync.ldap.user.searchfilter: "" + ranger.usersync.ldap.user.nameattribute: "{{ auth_provider.ldap_attr_user_name }}" + ranger.usersync.ldap.user.objectclass: "{{ auth_provider.ldap_object_class.user }}" # person + ranger.usersync.ldap.user.searchbase: "{{ auth_provider.ldap_search_base.user }}" + ranger.usersync.source.impl.class: org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder + ranger.usersync.user.searchenabled: 'true' + +#obsolete in 7.1.6 ranger.usersync.ldap.searchBase: "{{ auth_provider.ldap_base_dn }}" +#obsolete in 7.1.6 ranger.usersync.ldap.user.groupnameattribute: "{{ auth_provider.ldap_attr_group_name }}" +#obsolete in 7.1.6 ranger.usersync.group.search.first.enabled: 'true' +#obsolete in 7.1.6 ranger.usersync.group.searchenabled: 'true' + INFRA_SOLR: + SERVICEWIDE: + #set false for 7.1.7 since it doesnt build cm_solr + enable_ranger_authorization: false + SOLR_SERVER: + solr_graceful_stop_timeout: 5 + SPARK_ON_YARN: + SPARK_YARN_HISTORY_SERVER: + ssl_enabled: true + history_server_spnego_enabled: false + YARN: + SERVICEWIDE: + hadoop_secure_web_ui: false + JOBHISTORY: + mr2_jobhistory_java_heapsize: 134217728 + NODEMANAGER: + node_manager_java_heapsize: 134217728 + yarn_nodemanager_resource_memory_mb: 2048 + yarn_nodemanager_resource_cpu_vcores: 2 + RESOURCEMANAGER: + resource_manager_java_heapsize: 134217728 + yarn_scheduler_maximum_allocation_mb: 2048 + yarn_scheduler_maximum_allocation_vcores: 2 + ZOOKEEPER: + SERVICEWIDE: + zookeeper_datadir_autocreate: true + service_config_suppression_server_count_validator: true + SERVER: + zookeeper_server_java_heapsize: 134217728 + host_templates: + Masters: + ATLAS: [ATLAS_SERVER] + HBASE: [MASTER, REGIONSERVER, GATEWAY] + HDFS: [NAMENODE, SECONDARYNAMENODE, DATANODE] + HIVE: [HIVEMETASTORE, GATEWAY] + HIVE_ON_TEZ: [HIVESERVER2, GATEWAY] + INFRA_SOLR: [SOLR_SERVER] + KAFKA: [KAFKA_BROKER] + OZONE: [GATEWAY, OZONE_MANAGER, OZONE_RECON, STORAGE_CONTAINER_MANAGER, OZONE_DATANODE, S3_GATEWAY] + RANGER: [RANGER_ADMIN, RANGER_TAGSYNC, RANGER_USERSYNC] + SPARK_ON_YARN: [SPARK_YARN_HISTORY_SERVER, GATEWAY] + TEZ: [GATEWAY] + YARN: [RESOURCEMANAGER, NODEMANAGER, JOBHISTORY, GATEWAY] + ZOOKEEPER: [SERVER] + + Workers: + HBASE: [REGIONSERVER, GATEWAY] + HDFS: [DATANODE] + HIVE: [GATEWAY] + HIVE_ON_TEZ: [GATEWAY] + OZONE: [OZONE_DATANODE, GATEWAY] + TEZ: [GATEWAY] + SPARK_ON_YARN: [GATEWAY] + YARN: [NODEMANAGER, GATEWAY] + ZOOKEEPER: [SERVER] + +mgmt: + name: Cloudera Management Service + services: [ALERTPUBLISHER, EVENTSERVER, HOSTMONITOR, REPORTSMANAGER, SERVICEMONITOR] + security: + tls: true + configs: + HOSTMONITOR: + firehose_heapsize: 1073741824 + firehose_non_java_memory_bytes: 2147483648 + SERVICEMONITOR: + firehose_heapsize: 2147483648 + firehose_non_java_memory_bytes: 6442450944 + role_config_suppression_firehose_service_monitor_non_java_memory_role_validator: true + +hosts: + configs: + host_default_proc_memswap_thresholds: + warning: never + critical: never + host_memswap_thresholds: + warning: never + critical: never + host_config_suppression_agent_system_user_group_validator: true + host_config_suppression_memory_overcommitted_validator: true diff --git a/private-cloud/base/aws-iaas/config-template.yml b/private-cloud/base/aws-iaas/config-template.yml new file mode 100644 index 0000000..dda9179 --- /dev/null +++ b/private-cloud/base/aws-iaas/config-template.yml @@ -0,0 +1,25 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +--- + +name_prefix: "{{ mandatory }}" # Unique identifier for the deployment +infra_region: "us-east-2" +domain: "{{ name_prefix }}.cldr.example" # The deployment subdomain (private) managed by the local FreeIPA server +realm: "CLDR.DEPLOYMENT" # The Kerberos realm managed by the local FreeIPA server +common_password: "Example776" # For external services, e.g. FreeIPA, Postgres, TLS +admin_password: "Example776" # For Cloudera-related services, e.g. Cloudera Manager +deployment_tags: + deployment: "{{ name_prefix }}" + deploy-tool: cloudera-deploy diff --git a/private-cloud/base/aws-iaas/definition.yml b/private-cloud/base/aws-iaas/definition.yml new file mode 100644 index 0000000..177be9b --- /dev/null +++ b/private-cloud/base/aws-iaas/definition.yml @@ -0,0 +1,155 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +--- + +### Infrastructure + +infra_type: aws + +# Terraform +terraform: + state: + storage: local + create_remote_storage: False + s3_region: "{{ infra_region }}" + s3_bucket: "{{ name_prefix }}-tf-state" + +# SSH +public_key_id: "{{ name_prefix }}-pvc-base" +public_key_file: "{{ lookup('ansible.builtin.env', 'SSH_PUBLIC_KEY_FILE') }}" +public_key_text: "{{ lookup('ansible.builtin.file', lookup('ansible.builtin.env', 'SSH_PUBLIC_KEY_FILE')) }}" + +### Cluster + +kerberos_activated: True +encryption_activated: True +# tls_activated: True +# autotls: False + +jdk_version: 11 + +repositories: + - https://archive.cloudera.com/p/cdh7/7.1.9.0/parcels/ # Offical CDH 7.1.9.0 + +# Connect FreeIPA to Knox and Ranger +freeipa_activated: yes + +# FreeIPA client install on cluster nodes is done during pre_setup, so suppress +freeipa_enroll: no + +# FreeIPA realm settings +freeipa: + realm: "{{ realm }}" + ipaadmin_user: admin + ipaadmin_password: "{{ common_password }}" + +# Defer user and group creation to SSSD/FreeIPA registration +# Use when you dont need ansible to run "ca_server" +skip_user_group_init: no + +### Cloudera Manager + +cloudera_manager_version: 7.11.3 + +# General options +cloudera_manager_options: + CUSTOM_BANNER_HTML: "PVC Base Lab ({{ name_prefix }})" + SESSION_TIMEOUT: 43200 + PARCEL_DISTRIBUTE_RATE_LIMIT_KBS_PER_SECOND: 194560 + KRB_AUTH_ENABLE: "true" + +# Cloudera license +license_file: "{{ lookup('ansible.builtin.env', 'CDP_LICENSE_FILE') }}" + +# License options (this is due to a hardcoded tmp directory on the target/manager node) +license_local_tmp_path: /tmp/cloudera_license.txt + +### RDBMS + +database_type: postgresql +database_version: 12 +database_tls: true +database_default_password: "{{ common_password }}" + +### Red Hat FreeIPA + +krb5_kdc_type: Red Hat IPA +krb5_realm: "{{ freeipa.realm }}" +krb5_kdc_admin_user: "{{ freeipa.ipaadmin_user }}@{{ krb5_realm }}" +krb5_kdc_admin_password: "{{ freeipa.ipaadmin_password }}" +krb5_enc_types: "aes256-cts aes128-cts" +# TODO: This needs to either be explicit or rendered from a host with DNS resolution +# Cannot rely on localhost to have DNS access, e.g. private networking on EC2 +#krb5_kdc_host: "{{ lookup('community.general.dig', '_ldap._tcp.' + domain +'./SRV', 'flat=0', wantlist=True, fail_on_error=True) | map(attribute='target') | first }}" +krb5_kdc_host: "{{ groups['freeipa'] | first | default('') }}" + +### CM External Auth - FreeIPA as LDAP + +# Settings for FreeIPA sidecar deployment +cloudera_manager_external_auth: + provider: freeipa + external_first: no + external_only: no + +# Set in cloudera_manager.external_auth.tasks.main:26 +auth_provider: "{{ auth_providers[cloudera_manager_external_auth.provider] }}" + +base_dn: "dc={{ freeipa.realm.split('.') | map('lower') | join(',dc=') }}" +user_dn: "cn=users,cn=accounts,{{ base_dn }}" +group_dn: "cn=groups,cn=accounts,{{ base_dn }}" + +auth_providers: + freeipa: + ldap_url: "ldaps://{{ groups['freeipa'] | first | default('') }}" + ldap_base_dn: "{{ base_dn }}" + ldap_bind_user_dn: "uid=admin,{{ user_dn }}" + ldap_bind_password: "{{ freeipa.ipaadmin_password }}" + ldap_search_base: + user: "{{ user_dn }}" + group: "{{ group_dn }}" + ldap_object_class: + user: "person" + group: "posixgroup" + ldap_search_filter: # Used for mapping Services (Atlas, Ranger, etc.) to LDAP + user: "(&(sAMAccountName={0})(objectClass=person))" + member: "(&(member={0})(objectClass=group))" + # group: (&(member={0})(objectclass=posixgroup)(!(cn=admins))) + ldap_attribute: # Used for mapping CM to LDAP + user: uid # Set to (uid={0}) + member: member # defaults to (member={0}) + ldap_attr_user_name: "uid" + ldap_attr_group_name: "cn" + ldap_attr_group_membership: "member" + type: LDAP + +## TLS signing +skip_ipa_signing: no # This is the default +remote_ipa_server: "{{ groups['freeipa'] | first | default('') }}" + +tls_key_password: "{{ common_password }}" +tls_keystore_password: "{{ common_password }}" +tls_truststore_password: "{{ common_password }}" + +ca_server_attrs_root: + CN: 'SE-Root CA' +ca_server_attrs_intermediate: + CN: 'SE-Intermediate CA' + +# Vars from legacy cloudera-Deploy +use_auto_repo_mirror: no +use_default_cluster_definition: no +use_download_mirror: no +preload_cm_parcel_repo: yes +teardown_everything: yes diff --git a/private-cloud/base/aws-iaas/external_setup.yml b/private-cloud/base/aws-iaas/external_setup.yml new file mode 100644 index 0000000..07c0224 --- /dev/null +++ b/private-cloud/base/aws-iaas/external_setup.yml @@ -0,0 +1,62 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +--- + +- name: Set up CDP Private Cloud external prerequisites + hosts: localhost + connection: local + gather_facts: yes + vars: + definition_path: "./" + tasks: + - name: Set of deployment variables from definition.yml + ansible.builtin.include_role: + name: cloudera.exe.init_deployment + public: yes + when: init__completed is undefined + + - name: Prepare inventory for PvC Plays + ansible.builtin.include_role: + name: cloudera.exe.init_deployment + public: yes + tasks_from: prep_pvc.yml + tags: + - always + +- name: Init run tasks for all nodes + hosts: all + gather_facts: no + tasks: + - name: Group hosts by host template and TLS + ansible.builtin.include_role: + name: cloudera.cluster.deployment.groupby + + - name: Check connectivity to Inventory + ansible.builtin.wait_for_connection: + tags: + - always + +- name: Set up external prerequisites for CDP Private Cloud + ansible.builtin.import_playbook: cloudera.exe.pvc_base_prereqs_ext.yml + +# NOTE: Cluster hosts become unavilable after krb5 client setup +# This is a temporary workaround while we debug the source of the issue +- name: Reboot all cluster hosts + hosts: cluster + gather_facts: no + become: yes + tasks: + - name: Reboot all cluster hosts + ansible.builtin.reboot: \ No newline at end of file diff --git a/private-cloud/base/aws-iaas/group_vars/all.yml b/private-cloud/base/aws-iaas/group_vars/all.yml new file mode 100644 index 0000000..05f9b7f --- /dev/null +++ b/private-cloud/base/aws-iaas/group_vars/all.yml @@ -0,0 +1,20 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +--- + +# If needed, set the following variable to tell Ansible which key to use +# ansible_ssh_private_key_file: ~/.ssh/some-private-key + +ansible_ssh_private_key_file: "{{ lookup('ansible.builtin.env', 'SSH_PRIVATE_KEY_FILE') }}" diff --git a/private-cloud/base/aws-iaas/internal_setup.yml b/private-cloud/base/aws-iaas/internal_setup.yml new file mode 100644 index 0000000..45fd301 --- /dev/null +++ b/private-cloud/base/aws-iaas/internal_setup.yml @@ -0,0 +1,52 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +--- + +- name: Set up CDP Private Cloud internal prerequisites + hosts: localhost + connection: local + gather_facts: yes + vars: + definition_path: "./" + tasks: + - name: Set of deployment variables from definition.yml + ansible.builtin.include_role: + name: cloudera.exe.init_deployment + public: yes + when: init__completed is undefined + + - name: Prepare inventory for PvC Plays + ansible.builtin.include_role: + name: cloudera.exe.init_deployment + public: yes + tasks_from: prep_pvc.yml + tags: + - always + +- name: Init run tasks for all nodes + hosts: all + gather_facts: no + tasks: + - name: Group hosts by host template and TLS + ansible.builtin.include_role: + name: cloudera.cluster.deployment.groupby + + - name: Check connectivity to Inventory + ansible.builtin.wait_for_connection: + tags: + - always + +- name: Set up internal prerequisites for CDP Private Cloud + ansible.builtin.import_playbook: cloudera.exe.pvc_base_prereqs_int.yml diff --git a/private-cloud/base/aws-iaas/inventory.yml b/private-cloud/base/aws-iaas/inventory.yml new file mode 100644 index 0000000..d9987a4 --- /dev/null +++ b/private-cloud/base/aws-iaas/inventory.yml @@ -0,0 +1,16 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +plugin: cloud.terraform.terraform_provider +project_path: tf_proxied_cluster diff --git a/private-cloud/base/aws-iaas/pre_setup.yml b/private-cloud/base/aws-iaas/pre_setup.yml new file mode 100644 index 0000000..e57488c --- /dev/null +++ b/private-cloud/base/aws-iaas/pre_setup.yml @@ -0,0 +1,141 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +--- + +- name: Provision AWS infrastructure + hosts: localhost + connection: local + gather_facts: no + tags: infra + tasks: + - name: Set Terraform variables + ansible.builtin.template: + dest: tf_proxied_cluster/terraform.tfvars + src: infra.tfvars.j2 + + - name: Check for existing Terraform state file + ansible.builtin.stat: + path: tf_proxied_cluster/terraform.tfstate + register: __tf_state + + - name: Establish the infrastructure + cloud.terraform.terraform: + project_path: "tf_proxied_cluster/" + state: "present" + force_init: "{{ not __tf_state.stat.exists }}" + provider_upgrade: true + + - name: Establish the Ansible inventory from the infrastructure + ansible.builtin.meta: refresh_inventory + +- name: Ensure inventory readiness + hosts: all + gather_facts: no + tags: always + tasks: + - name: Check if host is ready + ansible.builtin.wait_for_connection: + timeout: 60 + +- name: Prepare and mount storage volumes + hosts: all + gather_facts: no + become: yes + tags: infra + tasks: + - name: Prepare storage volumes + when: storage_volumes is defined and storage_volumes | length > 0 + ansible.builtin.import_role: + name: cloudera.exe.mount + vars: + mount_volumes: "{{ storage_volumes }}" + mount_provider: "{{ infra_type }}" + +- name: Provision FreeIPA services + hosts: freeipa + gather_facts: yes + become: yes + tags: freeipa + vars: + vpc_cidr: "{{ lookup('cloud.terraform.tf_output', 'vpc', project_path='tf_proxied_cluster')['cidr_block'] }}" + module_defaults: + freeipa.ansible_freeipa.ipadnszone: + ipaadmin_password: "{{ freeipa.ipaadmin_password | default(lookup('env', 'IPA_PASSWORD', default='Undefined')) }}" + freeipa.ansible_freeipa.ipadnsrecord: + ipaadmin_password: "{{ freeipa.ipaadmin_password | default(lookup('env', 'IPA_PASSWORD', default='Undefined')) }}" + tasks: + - name: Set up the local FreeIPA server + ansible.builtin.import_role: + name: cloudera.exe.freeipa_server + vars: + ipaserver_hostname: "{{ inventory_hostname }}" + ipaserver_realm: "{{ freeipa.realm }}" + ipaserver_domain: "{{ domain }}" + ipaserver_no_host_dns: yes + ipaserver_setup_firewalld: no + ipaserver_setup_dns: yes + # See https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html + ipaserver_resolv_nameservers: ["{{ vpc_cidr | ansible.utils.ipmath(2) }}"] + ipaserver_auto_reverse: yes + ipaserver_no_forwarders: yes + ipaserver_forward_policy: only + ipaserver_recursion_acl_cidr: "{{ vpc_cidr }}" + ipaserver_copy_csr_to_controller: yes + ipaserver_ca_subject: "CN=CLDR-{{ name_prefix }}-RootCA,O={{ freeipa.realm }}" + ipaclient_mkhomedir: yes + # TODO Add test for these parameters - error in role is opaque + ipadm_password: "{{ freeipa.ipaadmin_password | default(lookup('env', 'IPA_PASSWORD', default='Undefined')) }}" + ipaadmin_password: "{{ freeipa.ipaadmin_password | default(lookup('env', 'IPA_PASSWORD', default='Undefined')) }}" + + - name: Create DNS zone for '{{ domain }}' in provisioned FreeIPA service + freeipa.ansible_freeipa.ipadnszone: + zone_name: "{{ domain }}" + dynamic_update: yes + allow_sync_ptr: yes + forward_policy: none + + - name: Create reverse DNS zone for '{{ vpc_cidr }}' in provisioned FreeIPA service + freeipa.ansible_freeipa.ipadnszone: + name_from_ip: "{{ vpc_cidr }}" + dynamic_update: yes + allow_sync_ptr: yes + +- name: Register all hosts with local FreeIPA server + hosts: all:!freeipa + gather_facts: yes + become: yes + tags: freeipa + tasks: + - name: Register host with local FreeIPA server + ansible.builtin.import_role: + name: cloudera.exe.freeipa_client + vars: + ipaserver_domain: "{{ domain }}" + ipaserver_realm: "{{ freeipa.realm }}" + ipa_hosts: "{{ groups['freeipa'] }}" + ipa_server_ips: "{{ groups['freeipa'] | map('extract', hostvars, ['ansible_host']) | list }}" + ipaadmin_password: "{{ freeipa.ipaadmin_password | default(lookup('ansible.builtin.env', 'IPA_PASSWORD', default='Undefined')) }}" + ipaadmin_principal: "{{ freeipa.ipaadmin_user | default(lookup('ansible.builtin.env', 'IPA_USER', default=omit)) }}" + enable_dns: yes + +- name: Establish supporting services resources + hosts: deployment + gather_facts: no + become: yes + tags: prereq + tasks: + - name: Establish additional cluster host requirements + ansible.builtin.import_role: + name: cluster_reqs diff --git a/private-cloud/base/aws-iaas/pre_teardown.yml b/private-cloud/base/aws-iaas/pre_teardown.yml new file mode 100644 index 0000000..c3c23a1 --- /dev/null +++ b/private-cloud/base/aws-iaas/pre_teardown.yml @@ -0,0 +1,31 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +--- + +- name: Deprovision the infrastructure resources + hosts: localhost + connection: local + gather_facts: no + tasks: + - name: Check for existing Terraform state file + ansible.builtin.stat: + path: tf_proxied_cluster/terraform.tfstate + register: __tf_state + + - name: Teardown the demo infrastructure via Terraform + when: __tf_state.stat.exists + cloud.terraform.terraform: + project_path: "tf_proxied_cluster/" + state: "absent" diff --git a/private-cloud/base/aws-iaas/roles/cluster_reqs/README.md b/private-cloud/base/aws-iaas/roles/cluster_reqs/README.md new file mode 100644 index 0000000..225dd44 --- /dev/null +++ b/private-cloud/base/aws-iaas/roles/cluster_reqs/README.md @@ -0,0 +1,38 @@ +Role Name +========= + +A brief description of the role goes here. + +Requirements +------------ + +Any pre-requisites that may not be covered by Ansible itself or the role should be mentioned here. For instance, if the role uses the EC2 module, it may be a good idea to mention in this section that the boto package is required. + +Role Variables +-------------- + +A description of the settable variables for this role should go here, including any variables that are in defaults/main.yml, vars/main.yml, and any variables that can/should be set via parameters to the role. Any variables that are read from other roles and/or the global scope (ie. hostvars, group vars, etc.) should be mentioned here as well. + +Dependencies +------------ + +A list of other roles hosted on Galaxy should go here, plus any details in regards to parameters that may need to be set for other roles, or variables that are used from other roles. + +Example Playbook +---------------- + +Including an example of how to use your role (for instance, with variables passed in as parameters) is always nice for users too: + + - hosts: servers + roles: + - { role: username.rolename, x: 42 } + +License +------- + +BSD + +Author Information +------------------ + +An optional section for the role authors to include contact information, or a website (HTML is not allowed). diff --git a/private-cloud/base/aws-iaas/roles/cluster_reqs/defaults/main.yml b/private-cloud/base/aws-iaas/roles/cluster_reqs/defaults/main.yml new file mode 100644 index 0000000..803ff7a --- /dev/null +++ b/private-cloud/base/aws-iaas/roles/cluster_reqs/defaults/main.yml @@ -0,0 +1,2 @@ +--- +# defaults file for cluster_reqs diff --git a/private-cloud/base/aws-iaas/roles/cluster_reqs/meta/main.yml b/private-cloud/base/aws-iaas/roles/cluster_reqs/meta/main.yml new file mode 100644 index 0000000..c572acc --- /dev/null +++ b/private-cloud/base/aws-iaas/roles/cluster_reqs/meta/main.yml @@ -0,0 +1,52 @@ +galaxy_info: + author: your name + description: your role description + company: your company (optional) + + # If the issue tracker for your role is not on github, uncomment the + # next line and provide a value + # issue_tracker_url: http://example.com/issue/tracker + + # Choose a valid license ID from https://spdx.org - some suggested licenses: + # - BSD-3-Clause (default) + # - MIT + # - GPL-2.0-or-later + # - GPL-3.0-only + # - Apache-2.0 + # - CC-BY-4.0 + license: license (GPL-2.0-or-later, MIT, etc) + + min_ansible_version: 2.1 + + # If this a Container Enabled role, provide the minimum Ansible Container version. + # min_ansible_container_version: + + # + # Provide a list of supported platforms, and for each platform a list of versions. + # If you don't wish to enumerate all versions for a particular platform, use 'all'. + # To view available platforms and versions (or releases), visit: + # https://galaxy.ansible.com/api/v1/platforms/ + # + # platforms: + # - name: Fedora + # versions: + # - all + # - 25 + # - name: SomePlatform + # versions: + # - all + # - 1.0 + # - 7 + # - 99.99 + + galaxy_tags: [] + # List tags for your role here, one per line. A tag is a keyword that describes + # and categorizes the role. Users find roles by searching for tags. Be sure to + # remove the '[]' above, if you add tags to this list. + # + # NOTE: A tag is limited to a single word comprised of alphanumeric characters. + # Maximum 20 tags per role. + +dependencies: [] + # List your role dependencies here, one per line. Be sure to remove the '[]' above, + # if you add dependencies to this list. diff --git a/private-cloud/base/aws-iaas/roles/cluster_reqs/tasks/RedHat-post.yml b/private-cloud/base/aws-iaas/roles/cluster_reqs/tasks/RedHat-post.yml new file mode 100644 index 0000000..da61684 --- /dev/null +++ b/private-cloud/base/aws-iaas/roles/cluster_reqs/tasks/RedHat-post.yml @@ -0,0 +1,19 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +- name: Install required Python38 libraries + ansible.builtin.pip: + name: "{{ python38_libraries }}" + state: present + executable: pip3.8 diff --git a/private-cloud/base/aws-iaas/roles/cluster_reqs/tasks/Ubuntu-pre.yml b/private-cloud/base/aws-iaas/roles/cluster_reqs/tasks/Ubuntu-pre.yml new file mode 100644 index 0000000..66ec3df --- /dev/null +++ b/private-cloud/base/aws-iaas/roles/cluster_reqs/tasks/Ubuntu-pre.yml @@ -0,0 +1,17 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +- name: Update apt cache + ansible.builtin.apt: + update_cache: yes diff --git a/private-cloud/base/aws-iaas/roles/cluster_reqs/tasks/default.yml b/private-cloud/base/aws-iaas/roles/cluster_reqs/tasks/default.yml new file mode 100644 index 0000000..102e442 --- /dev/null +++ b/private-cloud/base/aws-iaas/roles/cluster_reqs/tasks/default.yml @@ -0,0 +1,17 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +--- + +# no op diff --git a/private-cloud/base/aws-iaas/roles/cluster_reqs/tasks/main.yml b/private-cloud/base/aws-iaas/roles/cluster_reqs/tasks/main.yml new file mode 100644 index 0000000..4c5dab2 --- /dev/null +++ b/private-cloud/base/aws-iaas/roles/cluster_reqs/tasks/main.yml @@ -0,0 +1,64 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +--- + +- name: Gather host distribution details + ansible.builtin.setup: + gather_subset: distribution + +- name: Load distribution variables + ansible.builtin.include_vars: "{{ item }}" + with_first_found: + - "{{ ansible_facts['distribution'] }}-{{ ansible_facts['distribution_version'] }}.yml" + - "{{ ansible_facts['distribution'] }}-{{ ansible_facts['distribution_major_version'] }}.yml" + - "{{ ansible_facts['distribution'] }}.yml" + - "{{ ansible_facts['os_family'] }}-{{ ansible_facts['distribution_version'] }}.yml" + - "{{ ansible_facts['os_family'] }}-{{ ansible_facts['distribution_major_version'] }}.yml" + - "{{ ansible_facts['os_family'] }}.yml" + - "default.yml" + +- name: Run distribution pre-tasks + ansible.builtin.include_tasks: "{{ item }}" + with_first_found: + - "{{ ansible_facts['distribution'] }}-{{ ansible_facts['distribution_version'] }}-pre.yml" + - "{{ ansible_facts['distribution'] }}-{{ ansible_facts['distribution_major_version'] }}-pre.yml" + - "{{ ansible_facts['distribution'] }}-pre.yml" + - "{{ ansible_facts['os_family'] }}-{{ ansible_facts['distribution_version'] }}-pre.yml" + - "{{ ansible_facts['os_family'] }}-{{ ansible_facts['distribution_major_version'] }}-pre.yml" + - "{{ ansible_facts['os_family'] }}-pre.yml" + - "default.yml" + +- name: Install required system packages + when: system_packages + ansible.builtin.package: + name: "{{ system_packages }}" + state: present + +- name: Install required Python libraries + when: python_libraries + ansible.builtin.pip: + name: "{{ python_libraries }}" + state: present + +- name: Run distribution post-tasks + ansible.builtin.include_tasks: "{{ item }}" + with_first_found: + - "{{ ansible_facts['distribution'] }}-{{ ansible_facts['distribution_version'] }}-post.yml" + - "{{ ansible_facts['distribution'] }}-{{ ansible_facts['distribution_major_version'] }}-post.yml" + - "{{ ansible_facts['distribution'] }}-post.yml" + - "{{ ansible_facts['os_family'] }}-{{ ansible_facts['distribution_version'] }}-post.yml" + - "{{ ansible_facts['os_family'] }}-{{ ansible_facts['distribution_major_version'] }}-post.yml" + - "{{ ansible_facts['os_family'] }}-post.yml" + - "default.yml" diff --git a/private-cloud/base/aws-iaas/roles/cluster_reqs/tests/inventory b/private-cloud/base/aws-iaas/roles/cluster_reqs/tests/inventory new file mode 100644 index 0000000..878877b --- /dev/null +++ b/private-cloud/base/aws-iaas/roles/cluster_reqs/tests/inventory @@ -0,0 +1,2 @@ +localhost + diff --git a/private-cloud/base/aws-iaas/roles/cluster_reqs/tests/test.yml b/private-cloud/base/aws-iaas/roles/cluster_reqs/tests/test.yml new file mode 100644 index 0000000..37b6e3c --- /dev/null +++ b/private-cloud/base/aws-iaas/roles/cluster_reqs/tests/test.yml @@ -0,0 +1,5 @@ +--- +- hosts: localhost + remote_user: root + roles: + - cluster_reqs diff --git a/private-cloud/base/aws-iaas/roles/cluster_reqs/vars/RedHat.yml b/private-cloud/base/aws-iaas/roles/cluster_reqs/vars/RedHat.yml new file mode 100644 index 0000000..5a4f74f --- /dev/null +++ b/private-cloud/base/aws-iaas/roles/cluster_reqs/vars/RedHat.yml @@ -0,0 +1,27 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +system_packages: + - tree + - jq + - zip + #- postgresql-devel + #- gcc + #- platform-python-devel + - python38 + +python_libraries: [] + +python38_libraries: + - psycopg2-binary==2.9.5 diff --git a/private-cloud/base/aws-iaas/roles/cluster_reqs/vars/Ubuntu.yml b/private-cloud/base/aws-iaas/roles/cluster_reqs/vars/Ubuntu.yml new file mode 100644 index 0000000..0d0ecd8 --- /dev/null +++ b/private-cloud/base/aws-iaas/roles/cluster_reqs/vars/Ubuntu.yml @@ -0,0 +1,22 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +system_packages: + - tree + - jq + - zip + - python3-pip + +python_libraries: + - psycopg2-binary==2.9.5 \ No newline at end of file diff --git a/private-cloud/base/aws-iaas/roles/cluster_reqs/vars/default.yml b/private-cloud/base/aws-iaas/roles/cluster_reqs/vars/default.yml new file mode 100644 index 0000000..102e442 --- /dev/null +++ b/private-cloud/base/aws-iaas/roles/cluster_reqs/vars/default.yml @@ -0,0 +1,17 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +--- + +# no op diff --git a/private-cloud/base/aws-iaas/roles/cluster_reqs/vars/main.yml b/private-cloud/base/aws-iaas/roles/cluster_reqs/vars/main.yml new file mode 100644 index 0000000..ac1c250 --- /dev/null +++ b/private-cloud/base/aws-iaas/roles/cluster_reqs/vars/main.yml @@ -0,0 +1,2 @@ +--- +# vars file for cluster_reqs diff --git a/private-cloud/base/aws-iaas/summary.yml b/private-cloud/base/aws-iaas/summary.yml new file mode 100644 index 0000000..4cc24f1 --- /dev/null +++ b/private-cloud/base/aws-iaas/summary.yml @@ -0,0 +1,39 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +- name: Create a deployment summary document + hosts: localhost + connection: local + gather_facts: no + tasks: + - name: Create the deployment summary Markdown document + ansible.builtin.template: + src: "deployment.md.j2" + dest: "{{ name_prefix | upper }}-DEPLOYMENT.md" + + - name: Create the deployment summary HTML document + ansible.builtin.template: + src: "deployment.html.j2" + dest: "{{ name_prefix | upper }}-DEPLOYMENT.html" + +- name: Retrieve the FreeIPA CA certificate + hosts: freeipa + gather_facts: no + tasks: + - name: Retrieve the CA certificate + ansible.builtin.fetch: + src: "/etc/ipa/ca.crt" + dest: "{{ inventory_hostname }}.ca.crt" + flat: yes + run_once: yes diff --git a/private-cloud/base/aws-iaas/templates/deployment.html.j2 b/private-cloud/base/aws-iaas/templates/deployment.html.j2 new file mode 100644 index 0000000..4aacb09 --- /dev/null +++ b/private-cloud/base/aws-iaas/templates/deployment.html.j2 @@ -0,0 +1,150 @@ + + + + + + + + + + + {{ name_prefix}} - PVC Base Deployment + + + +
+
+

{{ name_prefix}} - PVC Base Deployment

+

A CDP Private Cloud Base cluster running on AWS IaaS.

+
+ + + + + + + + + + + + + + + + + +
Prefix{{ name_prefix }}
Domain{{ domain }}
Realm + {{ realm }}
SSH + {{ lookup('ansible.builtin.env', 'SSH_PUBLIC_KEY_FILE') | basename }}
+

Deployment Access

+

Enable the SSH tunnel to the deployment:

+ 
+  ssh -D 8157 -qCN {{ groups['jump_host'] | map('extract', hostvars, 'ansible_user') | first }}@{{ groups['jump_host'] | map('extract', hostvars, 'ansible_host') | first }}
+
+

Use a SOCKS5 proxy switcher in your browser to access the deployment endpoints. (An example is the + SwitchyOmega extension for Chrome-based browsers.)

+

In the proxy configuration, set Protocol to SOCKS5, Server to localhost, and + Port to 8157. Ensure the SOCKS5 proxy is active when clicking on the CDP Private Cloud Base UIs and + endpoints that you wish to access.

+

SSL Certificate Installation

+

Any SSL-enabled endpoint within the cluster will produce a SSL warning for the self-signed certificate issued by + the deployment's local FreeIPA server. If you wish to avoid this warning, install the CA certificate, {{ + groups['freeipa'] | first }}.ca.crt, for your browser and/or system.

+

Endpoints

+ + + + + + + + + + + +
Cloudera Managerhttps://{{ + groups['cloudera_manager'] | first }}:7183
FreeIPAhttps://{{ groups['freeipa'] | first + }}
+

Hosts

+ + + + + + + + + + {% for host in groups['all'] | sort | map('extract', hostvars) | list %} + + + + + + {% endfor %} + +
HostFQDNGroups
{{ host['inventory_hostname_short'] }}{{ host['inventory_hostname'] }}{{ host['group_names'] | join(', ') }}
+
+ + + \ No newline at end of file diff --git a/private-cloud/base/aws-iaas/templates/deployment.md.j2 b/private-cloud/base/aws-iaas/templates/deployment.md.j2 new file mode 100644 index 0000000..33401bf --- /dev/null +++ b/private-cloud/base/aws-iaas/templates/deployment.md.j2 @@ -0,0 +1,43 @@ +# {{ name_prefix}} - PVC Base Deployment + +> A CDP Private Cloud Base cluster running on AWS IaaS. + +* **Prefix:** {{ name_prefix }} +* **Domain:** {{ domain }} +* **Realm:** {{ realm }} +* **SSH:** {{ lookup('ansible.builtin.env', 'SSH_PUBLIC_KEY_FILE') | basename }} + +## Deployment Access + +Enable the SSH tunnel to the deployment: + +```bash +ssh -D 8157 -qCN {{ groups['jump_host'] | map('extract', hostvars, 'ansible_user') | first }}@{{ groups['jump_host'] | map('extract', hostvars, 'ansible_host') | first }} +``` + +Use a SOCKS5 proxy switcher in your browser to access the deployment endpoints. +(An example is the _SwitchyOmega_ extension for Chrome-based browsers.) + +In the proxy configuration, set _Protocol_ to `SOCKS5`, _Server_ to +`localhost`, and _Port_ to `8157`. Ensure the SOCKS5 proxy is active when +clicking on the CDP Private Cloud Base UIs and endpoints that you wish to access. + +## SSL Certificate Installation + +Any SSL-enabled endpoint within the cluster will produce a SSL warning for the +self-signed certificate issued by the deployment's local FreeIPA server. If you +wish to avoid this warning, install the CA certificate, `{{ groups['freeipa'] | first }}.ca.crt`, +for your browser and/or system. + +## Endpoints + +* [Cloudera Manager](https://{{ groups['cloudera_manager'] | first }}:7183) +* [FreeIPA](https://{{ groups['freeipa'] | first }}) + +## Hosts + +|Host|FQDN|Groups| +|---|---|---| +{% for host in groups['all'] | sort | map('extract', hostvars) | list %} +| {{ host['inventory_hostname_short'] }} | {{ host['inventory_hostname'] }} | {{ host['group_names'] | join(', ') }} +{% endfor %} diff --git a/private-cloud/base/aws-iaas/templates/infra.tfvars.j2 b/private-cloud/base/aws-iaas/templates/infra.tfvars.j2 new file mode 100644 index 0000000..d7f682a --- /dev/null +++ b/private-cloud/base/aws-iaas/templates/infra.tfvars.j2 @@ -0,0 +1,26 @@ +// Copyright 2023 Cloudera, Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +ssh_public_key_file = "{{ lookup('ansible.builtin.env', 'SSH_PUBLIC_KEY_FILE') }}" +domain = "{{ domain }}" +prefix = "{{ name_prefix}}" +region = "{{ infra_region }}" + +{% if tags is defined and tags is mapping %} +asset_tags = { +{% for k, v in tags.items() %} + {{ k }} = "{{ v }}" +{% endfor %} +} +{% endif %} diff --git a/private-cloud/base/aws-iaas/tf_bastion/main.tf b/private-cloud/base/aws-iaas/tf_bastion/main.tf new file mode 100644 index 0000000..7ab5f25 --- /dev/null +++ b/private-cloud/base/aws-iaas/tf_bastion/main.tf @@ -0,0 +1,66 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +terraform { + required_version = ">= 0.13" + required_providers { + aws = { + source = "hashicorp/aws", + version = ">= 4.60.0", + } + } +} + +locals { + security_group_name = var.security_group_name != "" ? var.security_group_name : "${var.prefix}-pvc-base-bastion" + instance_name = var.name != "" ? var.name : "${var.prefix}-pvc-base-bastion" +} + +data "aws_key_pair" "bastion" { + key_name = var.ssh_key_pair +} + +resource "aws_security_group" "bastion" { + name = local.security_group_name + description = "Allow SSH traffic" + vpc_id = var.vpc_id +} + +resource "aws_vpc_security_group_egress_rule" "bastion" { + security_group_id = aws_security_group.bastion.id + + cidr_ipv4 = "0.0.0.0/0" + ip_protocol = -1 + description = "All traffic" +} + +resource "aws_vpc_security_group_ingress_rule" "bastion" { + security_group_id = aws_security_group.bastion.id + + cidr_ipv4 = "0.0.0.0/0" + from_port = 22 + ip_protocol = "tcp" + to_port = 22 + description = "All SSH traffic" +} + +resource "aws_instance" "bastion" { + ami = var.image_id + instance_type = "t2.micro" + key_name = data.aws_key_pair.bastion.key_name + subnet_id = var.subnet_id + vpc_security_group_ids = [aws_security_group.bastion.id] + associate_public_ip_address = true + tags = { Name = local.instance_name } +} diff --git a/private-cloud/base/aws-iaas/tf_bastion/outputs.tf b/private-cloud/base/aws-iaas/tf_bastion/outputs.tf new file mode 100644 index 0000000..e20ba3f --- /dev/null +++ b/private-cloud/base/aws-iaas/tf_bastion/outputs.tf @@ -0,0 +1,23 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +output "host" { + value = aws_instance.bastion + description = "Bastion host" +} + +output "security_group" { + value = aws_security_group.bastion + description = "Bastion Security Group" +} diff --git a/private-cloud/base/aws-iaas/tf_bastion/variables.tf b/private-cloud/base/aws-iaas/tf_bastion/variables.tf new file mode 100644 index 0000000..002c59a --- /dev/null +++ b/private-cloud/base/aws-iaas/tf_bastion/variables.tf @@ -0,0 +1,61 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +variable "asset_tags" { + type = map(any) + default = {} + description = "Map of tags applied to all cloud-provider assets." +} + +variable "ssh_key_pair" { + type = string + description = "SSH key pair name" +} + +variable "prefix" { + type = string + description = "Deployment prefix for all cloud-provider assets." + + validation { + condition = length(var.prefix) < 8 || length(var.prefix) > 4 + error_message = "Valid length for prefix is between 4-7 characters." + } +} + +variable "vpc_id" { + type = string + description = "VPC ID" +} + +variable "subnet_id" { + type = string + description = "Subnet ID" +} + +variable "security_group_name" { + type = string + description = "Name of the bastion security group" + default = "" +} + +variable "image_id" { + type = string + description = "AMI image ID for the bastion" +} + +variable "name" { + type = string + description = "Name of the bastion instance" + default = "" +} diff --git a/private-cloud/base/aws-iaas/tf_hosts/main.tf b/private-cloud/base/aws-iaas/tf_hosts/main.tf new file mode 100644 index 0000000..e7333b3 --- /dev/null +++ b/private-cloud/base/aws-iaas/tf_hosts/main.tf @@ -0,0 +1,59 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +terraform { + required_version = ">= 0.13" + required_providers { + aws = { + source = "hashicorp/aws", + version = ">= 4.60.0", + } + } +} + +locals { + instance_name = var.name != "" ? var.name : "${var.prefix}-pvc-base" +} + +data "aws_ami" "pvc_base" { + filter { + name = "image-id" + values = [var.image_id] + } +} + +resource "aws_instance" "pvc_base" { + count = var.quantity + + key_name = var.ssh_key_pair + instance_type = var.instance_type + ami = data.aws_ami.pvc_base.id + + subnet_id = var.subnet_ids[count.index % length(var.subnet_ids)] + associate_public_ip_address = var.public_ip + + security_groups = var.security_groups + + root_block_device { + delete_on_termination = var.root_volume.delete_on_termination + volume_size = var.root_volume.volume_size + volume_type = var.root_volume.volume_type + } + + metadata_options { + http_tokens = data.aws_ami.pvc_base.imds_support == "v2.0" ? "required" : "optional" + } + + tags = merge(var.tags, { Name = format("%s-%02d", local.instance_name, count.index + var.offset + 1) }) +} diff --git a/private-cloud/base/aws-iaas/tf_hosts/outputs.tf b/private-cloud/base/aws-iaas/tf_hosts/outputs.tf new file mode 100644 index 0000000..413602d --- /dev/null +++ b/private-cloud/base/aws-iaas/tf_hosts/outputs.tf @@ -0,0 +1,18 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +output "hosts" { + value = aws_instance.pvc_base + description = "Hosts" +} diff --git a/private-cloud/base/aws-iaas/tf_hosts/variables.tf b/private-cloud/base/aws-iaas/tf_hosts/variables.tf new file mode 100644 index 0000000..24d3813 --- /dev/null +++ b/private-cloud/base/aws-iaas/tf_hosts/variables.tf @@ -0,0 +1,92 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +variable "tags" { + type = map(any) + default = {} + description = "Map of tags applied to all cloud-provider assets." +} + +variable "ssh_key_pair" { + type = string + description = "SSH key pair name" +} + +variable "prefix" { + type = string + description = "Deployment prefix for all cloud-provider assets." + + validation { + condition = length(var.prefix) < 8 || length(var.prefix) > 4 + error_message = "Valid length for prefix is between 4-7 characters." + } +} + +# ------- Instances ------- + +variable "name" { + type = string + description = "Instance name prefix" + default = "" +} + +variable "quantity" { + type = number + description = "Number of instances" + default = 1 +} + +variable "offset" { + type = number + description = "Number offset for instance name" + default = 0 +} + +variable "subnet_ids" { + type = list(string) + description = "List of subnet IDs to provision the instances" +} + +variable "security_groups" { + type = list(string) + description = "List of security group IDs to attach to the instances" +} + +variable "public_ip" { + type = bool + description = "Flag to assign public IP addresses to the hosts" + default = false +} + +variable "instance_type" { + type = string + description = "Instance type name for the hosts" + default = "t2.micro" +} + +variable "image_id" { + type = string + description = "AMI image ID for the hosts" +} + +variable "root_volume" { + type = object({ + delete_on_termination = optional(bool, true) + volume_size = optional(number, 100) + volume_type = optional(string) + }) + description = "Root volume details" + default = {} +} + diff --git a/private-cloud/base/aws-iaas/tf_network/main.tf b/private-cloud/base/aws-iaas/tf_network/main.tf new file mode 100644 index 0000000..f6c0d1e --- /dev/null +++ b/private-cloud/base/aws-iaas/tf_network/main.tf @@ -0,0 +1,171 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +terraform { + required_version = ">= 0.13" + required_providers { + aws = { + source = "hashicorp/aws", + version = ">= 4.60.0", + } + } +} + +data "aws_availability_zones" "pvc_base" { + state = "available" +} + +data "aws_vpc" "pvc_base" { + id = var.vpc_id +} + +data "aws_internet_gateway" "pvc_base" { + filter { + name = "attachment.vpc-id" + values = [data.aws_vpc.pvc_base.id] + } +} + +locals { + nat_name = var.nat_gateway_name != "" ? var.nat_gateway_name : "${var.prefix}-pvc-base-nat" + rt_public_name = var.public_route_table_name != "" ? var.public_route_table_name : "${var.prefix}-pvc-base-public" + rt_private_name = var.private_route_table_name != "" ? var.private_route_table_name : "${var.prefix}-pvc-base-private" + public_subnet_name = var.public_subnet_name != "" ? var.public_subnet_name : "${var.prefix}-pvc-base-public" + public_subnets = length(var.public_subnets) > 0 ? var.public_subnets : tolist([{ + name = "${local.public_subnet_name}-01", + cidr = cidrsubnet(data.aws_vpc.pvc_base.cidr_block, 8, 0), + az = data.aws_availability_zones.pvc_base.names[0], + tags = {} + }]) + private_subnet_name = var.private_subnet_name != "" ? var.private_subnet_name : "${var.prefix}-pvc-base-private" + private_subnets = length(var.private_subnets) > 0 ? var.private_subnets : tolist([{ + name = "${local.private_subnet_name}-01", + cidr = cidrsubnet(data.aws_vpc.pvc_base.cidr_block, 8, 1), + az = data.aws_availability_zones.pvc_base.names[0], + tags = {} + }]) + sg_intra_name = var.security_group_intra_name != "" ? var.security_group_intra_name : "${var.prefix}-pvc-base-intra" +} + +# ------- AWS Public Network infrastructure ------- + +# TODO Calculate a local for a single default public subnet or defer to the root tf_proxied_cluter module +# TODO Lookup AZ id by input (name) and if not found, use AZ value as ID + +# Public Subnets +resource "aws_subnet" "pvc_base_public" { + for_each = { for idx, subnet in local.public_subnets : idx => subnet } + + vpc_id = data.aws_vpc.pvc_base.id + cidr_block = each.value.cidr + map_public_ip_on_launch = true + availability_zone = each.value.az + tags = merge(each.value.tags, { Name = each.value.name }) +} + +resource "aws_route_table" "pvc_base_public" { + for_each = { for idx, subnet in local.public_subnets : idx => subnet } + + vpc_id = data.aws_vpc.pvc_base.id + + tags = { Name = format("%s-%02d", local.rt_public_name, index(local.public_subnets, each.value) + 1) } + + route { + cidr_block = "0.0.0.0/0" + gateway_id = data.aws_internet_gateway.pvc_base.id + } +} + +# Public Route Table Associations +resource "aws_route_table_association" "pvc_base_public" { + for_each = { for idx, subnet in aws_subnet.pvc_base_public : idx => subnet } + + subnet_id = each.value.id + route_table_id = aws_route_table.pvc_base_public[each.key].id +} + +# ------- AWS Private Networking infrastructure ------- + +# Network Gateways (NAT) +resource "aws_eip" "pvc_base" { + count = length(aws_subnet.pvc_base_public) + + tags = { Name = format("%s-%02d", local.nat_name, count.index + 1) } +} + +resource "aws_nat_gateway" "pvc_base" { + for_each = { for idx, subnet in aws_subnet.pvc_base_public : idx => subnet } + + subnet_id = each.value.id + allocation_id = aws_eip.pvc_base[each.key].id + connectivity_type = "public" + tags = { Name = format("%s-%02d", local.nat_name, each.key + 1) } +} + +# Private Subnets +resource "aws_subnet" "pvc_base_private" { + for_each = { for idx, subnet in local.private_subnets : idx => subnet } + + vpc_id = data.aws_vpc.pvc_base.id + cidr_block = each.value.cidr + map_public_ip_on_launch = false + availability_zone = each.value.az + tags = merge(each.value.tags, { Name = each.value.name }) +} + +# Private Route Tables +resource "aws_route_table" "pvc_base_private" { + for_each = { for idx, subnet in local.private_subnets : idx => subnet } + + vpc_id = data.aws_vpc.pvc_base.id + + tags = { Name = format("%s-%02d", local.rt_private_name, index(local.private_subnets, each.value) + 1) } + + route { + cidr_block = "0.0.0.0/0" + nat_gateway_id = aws_nat_gateway.pvc_base[(index(local.private_subnets, each.value) % length(aws_nat_gateway.pvc_base))].id + } +} + +# Private Route Table Associations +resource "aws_route_table_association" "pvc_base_private" { + for_each = { for idx, subnet in aws_subnet.pvc_base_private : idx => subnet } + + subnet_id = each.value.id + route_table_id = aws_route_table.pvc_base_private[each.key].id +} + +# ------- Security Groups ------- + +# Intra-cluster traffic +resource "aws_security_group" "pvc_base" { + vpc_id = data.aws_vpc.pvc_base.id + name = local.sg_intra_name + description = "Intra-cluster communication [${var.prefix}]" + tags = { Name = local.sg_intra_name } +} + +resource "aws_vpc_security_group_ingress_rule" "pvc_base" { + security_group_id = aws_security_group.pvc_base.id + description = "Self-reference ingress rule" + ip_protocol = -1 + referenced_security_group_id = aws_security_group.pvc_base.id +} + +resource "aws_vpc_security_group_egress_rule" "pvc_base" { + security_group_id = aws_security_group.pvc_base.id + description = "Self-reference egress rule" + ip_protocol = -1 + referenced_security_group_id = aws_security_group.pvc_base.id +} diff --git a/private-cloud/base/aws-iaas/tf_network/outputs.tf b/private-cloud/base/aws-iaas/tf_network/outputs.tf new file mode 100644 index 0000000..4710d7e --- /dev/null +++ b/private-cloud/base/aws-iaas/tf_network/outputs.tf @@ -0,0 +1,33 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +output "availability_zones" { + value = data.aws_availability_zones.pvc_base + description = "AWS Availability Zones" +} + +output "public_subnets" { + value = values(aws_subnet.pvc_base_public) + description = "Cluster public subnets" +} + +output "private_subnets" { + value = values(aws_subnet.pvc_base_private) + description = "Cluster private subnets" +} + +output "intra_cluster_security_group" { + value = aws_security_group.pvc_base + description = "Intra-cluster traffic Security Group" +} diff --git a/private-cloud/base/aws-iaas/tf_network/variables.tf b/private-cloud/base/aws-iaas/tf_network/variables.tf new file mode 100644 index 0000000..91278f4 --- /dev/null +++ b/private-cloud/base/aws-iaas/tf_network/variables.tf @@ -0,0 +1,106 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# ------- Required ------- + +variable "asset_tags" { + type = map(any) + default = {} + description = "Map of tags applied to all cloud-provider assets" +} + +variable "prefix" { + type = string + description = "Deployment prefix for all cloud-provider assets" + + validation { + condition = length(var.prefix) < 8 || length(var.prefix) > 4 + error_message = "Valid length for prefix is between 4-7 characters." + } +} + +variable "region" { + type = string + description = "AWS region" +} + +variable "vpc_id" { + type = string + description = "VPC ID" +} + +# ------- Network Resources ------- + +# Public Network infrastructure +variable "public_subnet_name" { + type = string + description = "Public Subnet name prefix" + default = "" +} + +variable "public_subnets" { + type = list(object({ + name = string + cidr = string + az = string + tags = map(string) + })) + + description = "List of Public Subnet details (name, CIDR, AZ, add'l tags)" + default = [] +} + +variable "public_route_table_name" { + type = string + description = "Public Route Table name prefix" + default = "" +} + +# Private Network infrastructure +variable "private_subnet_name" { + type = string + description = "Private Subnet name prefix" + default = "" +} + +variable "private_subnets" { + type = list(object({ + name = string + cidr = string + az = string + tags = map(string) + })) + + description = "List of Private Subnet details (name, CIDR, AZ, add'l tags)" + default = [] +} + +variable "nat_gateway_name" { + type = string + description = "NAT gateway name" + default = "" +} + +variable "private_route_table_name" { + type = string + description = "Private Route Table name prefix" + default = "" +} + +# Security Groups +variable "security_group_intra_name" { + type = string + description = "Security Group for intra-cluster communication" + default = "" +} diff --git a/private-cloud/base/aws-iaas/tf_proxied_cluster/main.tf b/private-cloud/base/aws-iaas/tf_proxied_cluster/main.tf new file mode 100644 index 0000000..b7b5cf2 --- /dev/null +++ b/private-cloud/base/aws-iaas/tf_proxied_cluster/main.tf @@ -0,0 +1,324 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# TODO Could use prefix groups to add GlobalVPN restrictions on a non-proxied cluster + +terraform { + required_version = ">= 0.13" + required_providers { + aws = { + source = "hashicorp/aws", + version = ">= 4.60.0", + } + ansible = { + source = "ansible/ansible" + version = ">= 1.0.0" + } + } +} + +provider "aws" { + region = var.region + default_tags { + tags = var.asset_tags + } +} + +locals { + # RedHat 8.6 + ami_user = "ec2-user" + ami_owners = ["309956199498"] + ami_filters = { + name = ["RHEL-8.6*"] + architecture = ["x86_64"] + } + domain = var.domain != "" ? var.domain : "${var.prefix}.pvc-base.cldr.example" + vpc_name = var.vpc_name != "" ? var.vpc_name : "${var.prefix}-pvc-base" + igw_name = var.igw_name != "" ? var.igw_name : "${var.prefix}-pvc-base-igw" +} + +# ------- AMI ------- + +data "aws_ami" "pvc_base" { + owners = local.ami_owners + most_recent = true + + dynamic "filter" { + for_each = local.ami_filters + + content { + name = filter.key + values = filter.value + } + } +} + +# ------- SSH ------- + +data "local_file" "ssh_public_key_file" { + filename = var.ssh_public_key_file +} + +resource "aws_key_pair" "pvc_base" { + key_name = "${var.prefix}-pvc-base" + public_key = data.local_file.ssh_public_key_file.content +} + +# ------- VPC ------- + +resource "aws_vpc" "pvc_base" { + cidr_block = var.vpc_cidr + tags = { Name = local.vpc_name } + instance_tenancy = "default" + enable_dns_support = true + enable_dns_hostnames = true +} + +resource "aws_internet_gateway" "pvc_base" { + vpc_id = aws_vpc.pvc_base.id + tags = { Name = local.igw_name } +} + +# ------- Network ------- + +module "cluster_network" { + source = "../tf_network" + + region = var.region + prefix = var.prefix + vpc_id = aws_vpc.pvc_base.id +} + +resource "aws_vpc_security_group_egress_rule" "pvc_base" { + security_group_id = module.cluster_network.intra_cluster_security_group.id + description = "All traffic" + ip_protocol = -1 + cidr_ipv4 = "0.0.0.0/0" + tags = { Name = "${var.prefix}-pvc-base-intra" } +} + +# ------- Bastion ------- + +module "bastion" { + source = "../tf_bastion" + depends_on = [aws_key_pair.pvc_base, module.cluster_network] + + prefix = var.prefix + name = "${var.prefix}-bastion" + image_id = data.aws_ami.pvc_base.image_id + vpc_id = aws_vpc.pvc_base.id + subnet_id = module.cluster_network.public_subnets[0].id + ssh_key_pair = aws_key_pair.pvc_base.key_name +} + +resource "aws_vpc_security_group_ingress_rule" "bastion" { + security_group_id = module.cluster_network.intra_cluster_security_group.id + description = "All Bastion traffic" + ip_protocol = -1 + referenced_security_group_id = module.bastion.security_group.id + tags = { Name = "${var.prefix}-pvc-base-bastion" } +} + +resource "aws_vpc_security_group_egress_rule" "bastion" { + security_group_id = module.cluster_network.intra_cluster_security_group.id + description = "All Bastion traffic" + ip_protocol = -1 + referenced_security_group_id = module.bastion.security_group.id + tags = { Name = "${var.prefix}-pvc-base-bastion" } +} + +resource "aws_vpc_security_group_ingress_rule" "cluster" { + security_group_id = module.bastion.security_group.id + description = "All PVC Base traffic" + ip_protocol = -1 + referenced_security_group_id = module.cluster_network.intra_cluster_security_group.id + tags = { Name = "${var.prefix}-pvc-base-cluster" } +} + +# TODO Review if redundant with existing egress +resource "aws_vpc_security_group_egress_rule" "cluster" { + security_group_id = module.bastion.security_group.id + description = "All PVC Base traffic" + ip_protocol = -1 + referenced_security_group_id = module.cluster_network.intra_cluster_security_group.id +} + +# ------- Cluster ------- + +module "masters" { + source = "../tf_hosts" + depends_on = [aws_key_pair.pvc_base, data.aws_ami.pvc_base] + + prefix = var.prefix + name = "${var.prefix}-master" + image_id = data.aws_ami.pvc_base.image_id + instance_type = "m5.4xlarge" + ssh_key_pair = aws_key_pair.pvc_base.key_name + subnet_ids = module.cluster_network.private_subnets[*].id + security_groups = [module.cluster_network.intra_cluster_security_group.id] + public_ip = false + + root_volume = { + volume_size = 250 + } +} + +module "workers" { + source = "../tf_hosts" + depends_on = [aws_key_pair.pvc_base, data.aws_ami.pvc_base] + + prefix = var.prefix + name = "${var.prefix}-worker" + quantity = 2 + image_id = data.aws_ami.pvc_base.image_id + instance_type = "c5.2xlarge" + ssh_key_pair = aws_key_pair.pvc_base.key_name + subnet_ids = module.cluster_network.private_subnets[*].id + security_groups = [module.cluster_network.intra_cluster_security_group.id] + public_ip = false + + root_volume = { + volume_size = 250 + } +} + +module "freeipa" { + source = "../tf_hosts" + depends_on = [aws_key_pair.pvc_base, data.aws_ami.pvc_base] + + prefix = var.prefix + name = "${var.prefix}-freeipa" + image_id = data.aws_ami.pvc_base.image_id + instance_type = "m5.large" # TODO Look up via region + ssh_key_pair = aws_key_pair.pvc_base.key_name + subnet_ids = module.cluster_network.private_subnets[*].id + security_groups = [module.cluster_network.intra_cluster_security_group.id] + public_ip = false +} + +# ------- Ansible Inventory ------- + +resource "ansible_group" "bastion" { + name = "jump_host" +} + +resource "ansible_group" "freeipa" { + name = "freeipa" +} + +resource "ansible_group" "db" { + name = "db_server" +} + +resource "ansible_group" "cm" { + name = "cloudera_manager" +} + +resource "ansible_group" "workers" { + name = "cluster_workers" + variables = { + host_template = "Workers" + } +} + +resource "ansible_group" "masters" { + name = "cluster_masters" + variables = { + host_template = "Masters" + } +} + +resource "ansible_group" "cluster" { + name = "cluster" + children = [ + ansible_group.masters.name, + ansible_group.workers.name + ] + variables = { + tls = "True" + } +} + +resource "ansible_group" "deployment" { + name = "deployment" + children = [ + ansible_group.cluster.name, + ansible_group.cm.name, + ansible_group.db.name, + ansible_group.freeipa.name + ] + variables = { + ansible_ssh_common_args = "-o ProxyCommand='ssh -i {{ lookup('ansible.builtin.env', 'SSH_PRIVATE_KEY_FILE') }} -o User=${local.ami_user} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -W %h:%p -q ${module.bastion.host.public_ip}'" + } +} + +resource "ansible_host" "masters" { + for_each = { for idx, host in module.masters.hosts : idx => host } + + name = format("%s.%s", each.value.tags["Name"], local.domain) + + groups = [ + ansible_group.masters.name, + ansible_group.cm.name, + ansible_group.db.name + ] + + variables = { + ansible_host = each.value.private_ip + ansible_user = local.ami_user + } +} + +resource "ansible_host" "workers" { + for_each = { for idx, host in module.workers.hosts : idx => host } + + name = format("%s.%s", each.value.tags["Name"], local.domain) + + groups = [ + ansible_group.workers.name + ] + + variables = { + ansible_host = each.value.private_ip + ansible_user = local.ami_user + } +} + +resource "ansible_host" "freeipa" { + for_each = { for idx, host in module.freeipa.hosts : idx => host } + + name = format("%s.%s", each.value.tags["Name"], local.domain) + + groups = [ + ansible_group.freeipa.name + ] + + variables = { + ansible_host = each.value.private_ip + ansible_user = local.ami_user + } +} + +resource "ansible_host" "bastion" { + name = format("%s.%s", module.bastion.host.tags["Name"], local.domain) + + groups = [ + ansible_group.bastion.name + ] + + variables = { + ansible_host = module.bastion.host.public_ip + ansible_user = local.ami_user + } +} diff --git a/private-cloud/base/aws-iaas/tf_proxied_cluster/outputs.tf b/private-cloud/base/aws-iaas/tf_proxied_cluster/outputs.tf new file mode 100644 index 0000000..3d84753 --- /dev/null +++ b/private-cloud/base/aws-iaas/tf_proxied_cluster/outputs.tf @@ -0,0 +1,49 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +output "ssh_key_pair" { + value = { + name = aws_key_pair.pvc_base.key_name + public_key = data.local_file.ssh_public_key_file + fingerprint = aws_key_pair.pvc_base.fingerprint + } + description = "SSH key" +} + +output "vpc" { + value = aws_vpc.pvc_base + description = "AWS VPC" +} + +output "availability_zones" { + value = module.cluster_network.availability_zones + description = "AWS Availability Zones" +} + +output "cluster" { + value = { + public_subnets = module.cluster_network.public_subnets + private_subnets = module.cluster_network.private_subnets + intra_cluster_security_group = module.cluster_network.intra_cluster_security_group + } + description = "Private Cloud cluster" +} + +output "bastion" { + value = { + host = module.bastion.host + security_group = module.bastion.security_group + } + description = "Bastion host" +} diff --git a/private-cloud/base/aws-iaas/tf_proxied_cluster/variables.tf b/private-cloud/base/aws-iaas/tf_proxied_cluster/variables.tf new file mode 100644 index 0000000..1a8187f --- /dev/null +++ b/private-cloud/base/aws-iaas/tf_proxied_cluster/variables.tf @@ -0,0 +1,68 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# ------- General and Provider Resources ------- + +variable "ssh_public_key_file" { + type = string + description = "Local SSH public key file" +} + +variable "prefix" { + type = string + description = "Deployment prefix for all cloud-provider assets" + + validation { + condition = length(var.prefix) < 8 || length(var.prefix) > 4 + error_message = "Valid length for prefix is between 4-7 characters." + } +} + +variable "region" { + type = string + description = "AWS region" +} + +variable "asset_tags" { + type = map(string) + default = {} + description = "Map of tags applied to all cloud-provider assets" +} + +# ------- Network Resources ------- + +variable "vpc_name" { + type = string + description = "VPC name" + default = "" +} + +# TODO Convert to list of CIDR blocks +variable "vpc_cidr" { + type = string + description = "VPC CIDR Block" + default = "10.10.0.0/16" +} + +variable "igw_name" { + type = string + description = "Internet Gateway name" + default = "" +} + +variable "domain" { + type = string + description = "Private subdomain for proxied hosts, e.g. pvc-base.cldr.example" + default = "" +} diff --git a/private-cloud/base/aws-iaas/validate_dns_lookups.yml b/private-cloud/base/aws-iaas/validate_dns_lookups.yml new file mode 100644 index 0000000..a6ed0a5 --- /dev/null +++ b/private-cloud/base/aws-iaas/validate_dns_lookups.yml @@ -0,0 +1,39 @@ +# Copyright 2023 Cloudera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +--- + +- name: Check that all hosts in inventory are reachable + hosts: all + gather_facts: no + tasks: + - name: Ping each host from controller + ping: + +- name: Check Forward and Reverse DNS lookups + hosts: localhost + connection: local + gather_facts: no + tasks: + - name: Print the hosts + debug: + var: groups['all'] + + - name: Forward lookup for all hosts + ansible.builtin.debug: + msg: + - "Forward lookup DNS for {{ item }} is {{ lookup('community.general.dig', item , qtype='A') }}" + loop: "{{ groups['all'] }}" + + # TODO Register forward lookup IP and do reverse lookup