Skip to content

Add Terraform deployment engine for cloud resources #56

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 33 commits into from
Feb 25, 2022
Merged

Add Terraform deployment engine for cloud resources #56

merged 33 commits into from
Feb 25, 2022

Conversation

@jimright
Copy link
Contributor

@jimright jimright commented Oct 5, 2021

Added option for Terraform deployment of the cloud resources at the infrastructure and platform run-levels.
Features:

  • A parameter infra_deployment_engine used to determine the choice of deployment engine (pure-Ansible or Terraform) for cloud resources.
  • Terraform deployment is driven by Ansible, which creates necessary .tf resource files from Jinja templates and runs terraform apply\destroy.
  • Currently only supports the AWS infra_type:
    • Creates network, storage and compute resources at the infra level
    • Creates the AWS IAM policies and roles at the plat level
  • Support for local and remote_s3 (S3 bucket for storage; DynamoDB for state locking) backend storage of the Terraform state.
  • A timestamped artefact directory with a copy of the Terraform files created at each execution of cloudera-deploy.

Draft PR has a debug_terraform bool variable. This puts pause points before the main terraform steps allowing for review. These will be cleaned before the final PR.

@jimright jimright force-pushed the feature/terraform_infra branch 2 times, most recently from a5ea4db to ff2efd7 Compare October 5, 2021 20:09
@jimright jimright mentioned this pull request Oct 5, 2021
Merged
@jimright jimright marked this pull request as ready for review October 27, 2021 14:04
@wmudge wmudge requested review from Chaffelson and wmudge October 29, 2021 17:28
@wmudge wmudge added the enhancement MINOR - New feature or enhancement in the CHANGELOG label Nov 4, 2021
@jimright jimright force-pushed the feature/terraform_infra branch from f7d5ce9 to cd09c7c Compare December 1, 2021 19:38
@jimright
Copy link
Contributor Author

jimright commented Dec 2, 2021

Have rebased the code in this PR following the 1.5.1 release and retested that L0 and L1 deployment and teardown with Terraform work ok.

Chaffelson
Chaffelson suggested changes Dec 10, 2021
View reviewed changes
Copy link
Contributor

@Chaffelson Chaffelson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. Tested successfully on L0 and L1 networking with local state, will set up remote state or wait for you to implement automatic remote state in cloudera-deploy to go with it.

@jimright
Copy link
Contributor Author

Hey @Chaffelson

Thanks for looking through this.
I've changed and updated cloudera.exe based on your comments.
See above replies to your comments for specific details of changes.

Together with the updates to cloudera-deploy, this should be ready to retest now.

@jimright jimright requested a review from Chaffelson December 13, 2021 16:01
@jimright
Copy link
Contributor Author

@Chaffelson @wmudge,

Have updated the cloudera.exe Terraform PR following the release of the AWS provider.
This new provider version had some breaking changes and deprecations - listed here - so I've handled those and tested to confirm everything still works.

Thanks,
Jim

@jimright
Add Terraform templates for AWS infra and roles
d094656 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Refactor name of deployment engine variable
e571d93 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Update infrastructure and platform roles to execute Terraform tasks
30117ee 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Add Terraform templates for AWS infra and roles
1635ac2 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Add Terraform templates for AWS infra and roles
e2d90a9 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Cleanup of Terraform tasks
df05e26 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Update infrastructure and platform roles for teardown of Terraform re…
13fba68 
…sources

Signed-off-by: Jim Enright <[email protected]>
@jimright
Cleanup of Terraform tasks and variables
39c3ca4 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Push Terraform template and workspace variables to common
3eb0cb7 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Add S3 remote state for Terraform and code cleanup
8d6246b 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Cleanup of Terraform tasks and variables
ccbd1c8 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Create timestamped artefact of the generated Terraform files
45c4d39 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Refactor download of AWS policy docs to initialize
1304e19 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Use tempdir for Terraform workspace when remote state
ddfdd80 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Revert "Use tempdir for Terraform workspace when remote state"
cdcaed2 
This reverts commit 1360527.

Signed-off-by: Jim Enright <[email protected]>
@jimright
Add validation of variables used for Terraform
26ce3f0 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Cleanup of Terraform tasks and variables
ca8f4e9 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Update Terraform infra templates for L0, L1 and L2 changes
78c76fa 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Update Terraform infra templates for L0, L1 and L2 changes
a55260a 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Remove AWS DynamoDB reference from Terraform templates
43f3809 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Correct AWS CDP IDBroker role's policy definitions for Terraform temp…
275ad9f 
…lates

Signed-off-by: Jim Enright <[email protected]>
@jimright
Update Terraform platform for L0, L1 and L2 changes
efc4739 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Update Terraform platform for L0, L1 and L2 changes
35cdc5d 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Cleanup of Terraform tasks and variables
e4d4338 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Add DNS support to Terraform created VPC
ddac819 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Remove public IP assignment from private subnets
09c08a1 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Cleanup of Terraform tasks and variables
992122b 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Update Terraform infra and plat following PR feedback
8b9c340 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Populate dictionary for Static Inventory artefact during Terraform flow
128dac7 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Cleanup of Terraform tasks
14986d9 
Signed-off-by: Jim Enright <[email protected]>
@jimright
Update TF templates following AWS provider update
6a13b52 
Signed-off-by: Jim Enright <[email protected]>
@jimright jimright force-pushed the feature/terraform_infra branch from 20383f0 to 6a13b52 Compare February 24, 2022 19:56
wmudge
wmudge requested changes Feb 24, 2022
View reviewed changes
roles/infrastructure/tasks/setup_terraform.yml
when: infra__create_utility_service
block:
- name: Discover the Utility Instance details
community.aws.ec2_instance_info:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use output variables in TF?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above, I will test this and submit a separate PR if it looks good.

@jimright jimright requested a review from wmudge February 25, 2022 18:05
@jimright
Copy link
Contributor Author

@wmudge.

Thanks for looking through the PR. I have updated the code to address most of the issues you spotted. Also retested the AWS Terraform deployment and teardown to make sure that still works.

For things like using terraform output instead of Ansible-based discovery, I will test this separately and create a PR if all goes well.

Jim

@wmudge wmudge merged commit 5c3b33b into cloudera-labs:devel Feb 25, 2022
Chaffelson pushed a commit to Chaffelson/cloudera.exe that referenced this pull request Mar 9, 2022
@jimright @Chaffelson
Add Terraform deployment engine for cloud resources (cloudera-labs#56)
3a586cc 
* Add Terraform templates for AWS infra and roles
* Update infrastructure and platform roles to execute Terraform tasks
* Push Terraform template and workspace variables to common
* Add S3 remote state for Terraform
* Create timestamped artefact of the generated Terraform files
* Refactor download of AWS policy docs to initialize
* Add validation of variables used for Terraform
* Update Terraform infra templates for L0, L1 and L2 networking

Signed-off-by: Jim Enright <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wmudge wmudge wmudge approved these changes

+1 more reviewer

@Chaffelson Chaffelson Chaffelson approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

enhancement MINOR - New feature or enhancement in the CHANGELOG

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jimright @wmudge @Chaffelson