From 90c12e6df6dba6f9d17e42e040ad7b067f63a897 Mon Sep 17 00:00:00 2001 From: Webster Mudge Date: Tue, 26 Aug 2025 12:06:46 -0400 Subject: [PATCH 1/3] Remove legacy testing framework Signed-off-by: Webster Mudge --- .../integration_config.yml.template | 55 ----------------- tests/integration/requirements.txt | 29 --------- tests/integration/targets/exec_cdp/aliases | 15 ----- .../targets/exec_cdp/meta/main.yml | 17 ------ .../targets/exec_cdp/tasks/main.yml | 50 ---------------- .../integration/targets/light_duty_l1/aliases | 17 ------ .../targets/light_duty_l1/tasks/main.yml | 59 ------------------- .../targets/light_duty_l1/vars/cdp.yml | 33 ----------- .../targets/medium_duty_l1/aliases | 17 ------ .../targets/medium_duty_l1/tasks/main.yml | 45 -------------- .../targets/medium_duty_l1/vars/cdp.yml | 33 ----------- tests/integration/targets/setup_infra/aliases | 15 ----- .../targets/setup_infra/meta/main.yml | 18 ------ .../targets/setup_infra/tasks/main.yml | 43 -------------- .../targets/setup_infra/vars/infra.yml | 13 ---- .../targets/teardown_infra/aliases | 15 ----- .../targets/teardown_infra/meta/main.yml | 18 ------ .../targets/teardown_infra/tasks/main.yml | 43 -------------- .../targets/teardown_infra/vars/infra.yml | 26 -------- .../integration/test_molecule_integration.py | 16 ++--- 20 files changed, 9 insertions(+), 568 deletions(-) delete mode 100644 tests/integration/integration_config.yml.template delete mode 100644 tests/integration/requirements.txt delete mode 100644 tests/integration/targets/exec_cdp/aliases delete mode 100644 tests/integration/targets/exec_cdp/meta/main.yml delete mode 100644 tests/integration/targets/exec_cdp/tasks/main.yml delete mode 100644 tests/integration/targets/light_duty_l1/aliases delete mode 100644 tests/integration/targets/light_duty_l1/tasks/main.yml delete mode 100644 tests/integration/targets/light_duty_l1/vars/cdp.yml delete mode 100644 tests/integration/targets/medium_duty_l1/aliases delete mode 100644 tests/integration/targets/medium_duty_l1/tasks/main.yml delete mode 100644 tests/integration/targets/medium_duty_l1/vars/cdp.yml delete mode 100644 tests/integration/targets/setup_infra/aliases delete mode 100644 tests/integration/targets/setup_infra/meta/main.yml delete mode 100644 tests/integration/targets/setup_infra/tasks/main.yml delete mode 100644 tests/integration/targets/setup_infra/vars/infra.yml delete mode 100644 tests/integration/targets/teardown_infra/aliases delete mode 100644 tests/integration/targets/teardown_infra/meta/main.yml delete mode 100644 tests/integration/targets/teardown_infra/tasks/main.yml delete mode 100644 tests/integration/targets/teardown_infra/vars/infra.yml diff --git a/tests/integration/integration_config.yml.template b/tests/integration/integration_config.yml.template deleted file mode 100644 index 3f0b7089..00000000 --- a/tests/integration/integration_config.yml.template +++ /dev/null @@ -1,55 +0,0 @@ ---- - -# Copyright 2025 Cloudera, Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# integration_config.yml - -# Minimal cloudera.exe configuration to construct a CDP testing environment -# Will recursively merge and overwrite any variables defined within a Target -# (i.e. vars/main.yml) - -# You can use this file to create specific conditions for testing that are -# outside of a given target's configuration. Just be mindful that any changes -# here will proprogate to all targets, so depending on your testing and development, -# you might want to limit execution to specified targets. - -# Specify any environment variables for modules and libraries like CDPCLI, AWS, etc. -env_vars: - #AWS_PROFILE: my-aws-profile - AWS_ACCESS_KEY_ID: "foobarbaz" - AWS_SECRET_ACCESS_KEY: "1234567890" - #CDP_PROFILE: my-cdp-profile - CDP_ACCESS_KEY_ID: "foobarbaz" - CDP_PRIVATE_KEY: "1234567890" - -# Parameters used by all functions of cloudera.exe -globals: - name_prefix: t01 - admin_password: My$Very%Secret5Password9 - infra_type: aws - region: us-east-2 - ssh: - public_key_id: my-uploaded-ssh-key - tags: - deploy-tool: ansible-test - artifacts: - create_deployment_details: no - directory: "{{ playbook_dir }}/artifacts" - -# Default to cloud-only access patterns -infra: - storage: - name: "{{ globals.name_prefix }}-platform-test" - extra_ports: [22, 443] # Cloud-only access diff --git a/tests/integration/requirements.txt b/tests/integration/requirements.txt deleted file mode 100644 index 75de3563..00000000 --- a/tests/integration/requirements.txt +++ /dev/null @@ -1,29 +0,0 @@ -# Copyright 2025 Cloudera, Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# CDPCLI / cdpy -git+https://github.com/cloudera-labs/cdpy@main#egg=cdpy - -# AWS SDKs -awscli>=1.18.222 -boto>=2.49.0 -botocore>=1.12.249 -boto3>=1.9.249 - -# Ansible -jmespath # community.general.json_query -netaddr # ansible.netcommon.ipaddr - -# Upstream dependencies -docutils==0.14 # Required by cdpcli-beta 0.9.59 diff --git a/tests/integration/targets/exec_cdp/aliases b/tests/integration/targets/exec_cdp/aliases deleted file mode 100644 index aba01a22..00000000 --- a/tests/integration/targets/exec_cdp/aliases +++ /dev/null @@ -1,15 +0,0 @@ -# Copyright 2025 Cloudera, Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -hidden diff --git a/tests/integration/targets/exec_cdp/meta/main.yml b/tests/integration/targets/exec_cdp/meta/main.yml deleted file mode 100644 index d76244cd..00000000 --- a/tests/integration/targets/exec_cdp/meta/main.yml +++ /dev/null @@ -1,17 +0,0 @@ ---- - -# Copyright 2025 Cloudera, Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -dependencies: ["cloudera.exe.common"] diff --git a/tests/integration/targets/exec_cdp/tasks/main.yml b/tests/integration/targets/exec_cdp/tasks/main.yml deleted file mode 100644 index 37d78c28..00000000 --- a/tests/integration/targets/exec_cdp/tasks/main.yml +++ /dev/null @@ -1,50 +0,0 @@ ---- -# Copyright 2025 Cloudera, Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -- name: Validate the CDP Public Cloud environment configuration - ansible.builtin.import_role: - name: cloudera.exe.platform - tasks_from: validate.yml - environment: "{{ env_vars }}" - -- name: Install the CDP Public Cloud environment - when: install | bool - block: - - name: Initialize the CDP Public Cloud environment for setup - ansible.builtin.import_role: - name: cloudera.exe.platform - tasks_from: initialize_setup.yml - environment: "{{ env_vars }}" - - - name: Set up the CDP Public Cloud environment - ansible.builtin.import_role: - name: cloudera.exe.platform - tasks_from: setup.yml - environment: "{{ env_vars }}" - -- name: Teardown the CDP Public Cloud environment - when: not install | bool - block: - - name: Initialize the CDP Public Cloud environment for teardown - ansible.builtin.import_role: - name: cloudera.exe.platform - tasks_from: initialize_teardown.yml - environment: "{{ env_vars }}" - - - name: Tear down the CDP Public Cloud environment - ansible.builtin.import_role: - name: cloudera.exe.platform - tasks_from: teardown.yml - environment: "{{ env_vars }}" diff --git a/tests/integration/targets/light_duty_l1/aliases b/tests/integration/targets/light_duty_l1/aliases deleted file mode 100644 index 72664b00..00000000 --- a/tests/integration/targets/light_duty_l1/aliases +++ /dev/null @@ -1,17 +0,0 @@ -# Copyright 2025 Cloudera, Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -skip/python2 -setup/once/setup_infra -needs/target/exec_cdp diff --git a/tests/integration/targets/light_duty_l1/tasks/main.yml b/tests/integration/targets/light_duty_l1/tasks/main.yml deleted file mode 100644 index 750372db..00000000 --- a/tests/integration/targets/light_duty_l1/tasks/main.yml +++ /dev/null @@ -1,59 +0,0 @@ ---- -# Copyright 2025 Cloudera, Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -- name: Read CDP Public Cloud environment variables - ansible.builtin.include_vars: - file: cdp.yml - name: __cdp - tags: always - -- name: Combine CDP Public Cloud environment variables with runtime variables - ansible.builtin.set_fact: - globals: "{{ __cdp.globals | default({}) | combine(globals | default({}), recursive=True) }}" - env: "{{ __cdp.env | default({}) | combine(env | default({}), recursive=True) }}" - tags: always - -- name: Set up the CDP Public Cloud environment - ansible.builtin.import_role: - name: exec_cdp - vars: - install: true - tags: setup - -- name: Set up the CDP Public Cloud environment again (idempotent) - ansible.builtin.import_role: - name: exec_cdp - vars: - install: true - tags: setup - -- name: Run test tasks - ansible.builtin.debug: - msg: "No tests defined" - tags: test - -- name: Tear down the CDP Public Cloud environment - ansible.builtin.import_role: - name: exec_cdp - vars: - install: false - tags: teardown - -- name: Tear down the CDP Public Cloud environment (idempotent) - ansible.builtin.import_role: - name: exec_cdp - vars: - install: false - tags: teardown diff --git a/tests/integration/targets/light_duty_l1/vars/cdp.yml b/tests/integration/targets/light_duty_l1/vars/cdp.yml deleted file mode 100644 index 5051909e..00000000 --- a/tests/integration/targets/light_duty_l1/vars/cdp.yml +++ /dev/null @@ -1,33 +0,0 @@ ---- - -# Copyright 2025 Cloudera, Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Light-duty Datalake -# Level 1 networking (public/private) - -env: - name: "{{ globals.name_prefix }}-{{ globals.infra_type}}-lgt-l1-test" - cdp: - credential: - name: "{{ globals.name_prefix }}-{{ globals.infra_type }}-lgt-l1-test" - datalake: - scale: LIGHT_DUTY # same as env.enable_ha=False - tunnel: true - public_endpoint_access: true - workload_analytics: false - teardown: - delete_credential: true - azure: - single_resource_group: true diff --git a/tests/integration/targets/medium_duty_l1/aliases b/tests/integration/targets/medium_duty_l1/aliases deleted file mode 100644 index 72664b00..00000000 --- a/tests/integration/targets/medium_duty_l1/aliases +++ /dev/null @@ -1,17 +0,0 @@ -# Copyright 2025 Cloudera, Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -skip/python2 -setup/once/setup_infra -needs/target/exec_cdp diff --git a/tests/integration/targets/medium_duty_l1/tasks/main.yml b/tests/integration/targets/medium_duty_l1/tasks/main.yml deleted file mode 100644 index f3dcc4d9..00000000 --- a/tests/integration/targets/medium_duty_l1/tasks/main.yml +++ /dev/null @@ -1,45 +0,0 @@ ---- -# Copyright 2025 Cloudera, Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -- name: Read CDP Public Cloud environment variables - ansible.builtin.include_vars: - file: cdp.yml - name: __cdp - tags: always - -- name: Combine CDP Public Cloud environment variables with runtime variables - ansible.builtin.set_fact: - globals: "{{ __cdp.globals | default({}) | combine(globals | default({}), recursive=True) }}" - env: "{{ __cdp.env | default({}) | combine(env | default({}), recursive=True) }}" - tags: always - -- name: Set up the CDP Public Cloud environment - ansible.builtin.import_role: - name: exec_cdp - vars: - install: true - tags: setup - -- name: Run test tasks - ansible.builtin.debug: - msg: "No tests defined" - tags: test - -- name: Tear down the CDP Public Cloud environment - ansible.builtin.import_role: - name: exec_cdp - vars: - install: false - tags: teardown diff --git a/tests/integration/targets/medium_duty_l1/vars/cdp.yml b/tests/integration/targets/medium_duty_l1/vars/cdp.yml deleted file mode 100644 index 62d93678..00000000 --- a/tests/integration/targets/medium_duty_l1/vars/cdp.yml +++ /dev/null @@ -1,33 +0,0 @@ ---- - -# Copyright 2025 Cloudera, Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Medium-duty (HA) Datalake -# Level 1 networking (public/private) - -env: - name: "{{ globals.name_prefix }}-{{ globals.infra_type}}-md-l1-test" - cdp: - credential: - name: "{{ globals.name_prefix }}-{{ globals.infra_type }}-md-l1-test" - datalake: - scale: MEDIUM_DUTY_HA # same as env.enable_ha=True - tunnel: true - public_endpoint_access: true - workload_analytics: false - teardown: - delete_credential: true - azure: - single_resource_group: true diff --git a/tests/integration/targets/setup_infra/aliases b/tests/integration/targets/setup_infra/aliases deleted file mode 100644 index 9c11294f..00000000 --- a/tests/integration/targets/setup_infra/aliases +++ /dev/null @@ -1,15 +0,0 @@ -# Copyright 2025 Cloudera, Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -skip/python2 diff --git a/tests/integration/targets/setup_infra/meta/main.yml b/tests/integration/targets/setup_infra/meta/main.yml deleted file mode 100644 index f4bfc0bb..00000000 --- a/tests/integration/targets/setup_infra/meta/main.yml +++ /dev/null @@ -1,18 +0,0 @@ ---- - -# Copyright 2025 Cloudera, Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -dependencies: - - cloudera.exe.common diff --git a/tests/integration/targets/setup_infra/tasks/main.yml b/tests/integration/targets/setup_infra/tasks/main.yml deleted file mode 100644 index d79a5b9b..00000000 --- a/tests/integration/targets/setup_infra/tasks/main.yml +++ /dev/null @@ -1,43 +0,0 @@ ---- -# Copyright 2025 Cloudera, Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -- name: Read CDP Public Cloud environment infrastructure variables - ansible.builtin.include_vars: - file: infra.yml - name: __infra - -- name: Combine with CDP Public Cloud environment runtime variables - ansible.builtin.set_fact: - globals: "{{ __infra.globals | default({}) | combine(globals | default({}), recursive=True) }}" - infra: "{{ __infra.infra | default({}) | combine(infra | default({}), recursive=True) }}" - env: "{{ __infra.env | default({}) | combine(env | default({}), recursive=True) }}" - -- name: Validate CDP Public Cloud environment variables - ansible.builtin.import_role: - name: cloudera.exe.infrastructure - tasks_from: validate.yml - environment: "{{ env_vars }}" - -- name: Initialize CDP Public Cloud environment setup variables - ansible.builtin.import_role: - name: cloudera.exe.infrastructure - tasks_from: initialize_setup.yml - environment: "{{ env_vars }}" - -- name: Setup CDP Public Cloud environment infrastructure - ansible.builtin.import_role: - name: cloudera.exe.infrastructure - tasks_from: setup.yml - environment: "{{ env_vars }}" diff --git a/tests/integration/targets/setup_infra/vars/infra.yml b/tests/integration/targets/setup_infra/vars/infra.yml deleted file mode 100644 index aa8790a6..00000000 --- a/tests/integration/targets/setup_infra/vars/infra.yml +++ /dev/null @@ -1,13 +0,0 @@ -# Copyright 2025 Cloudera, Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. diff --git a/tests/integration/targets/teardown_infra/aliases b/tests/integration/targets/teardown_infra/aliases deleted file mode 100644 index 9c11294f..00000000 --- a/tests/integration/targets/teardown_infra/aliases +++ /dev/null @@ -1,15 +0,0 @@ -# Copyright 2025 Cloudera, Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -skip/python2 diff --git a/tests/integration/targets/teardown_infra/meta/main.yml b/tests/integration/targets/teardown_infra/meta/main.yml deleted file mode 100644 index f4bfc0bb..00000000 --- a/tests/integration/targets/teardown_infra/meta/main.yml +++ /dev/null @@ -1,18 +0,0 @@ ---- - -# Copyright 2025 Cloudera, Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -dependencies: - - cloudera.exe.common diff --git a/tests/integration/targets/teardown_infra/tasks/main.yml b/tests/integration/targets/teardown_infra/tasks/main.yml deleted file mode 100644 index 726b1abe..00000000 --- a/tests/integration/targets/teardown_infra/tasks/main.yml +++ /dev/null @@ -1,43 +0,0 @@ ---- -# Copyright 2025 Cloudera, Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -- name: Read CDP Public Cloud environment infrastructure variables - ansible.builtin.include_vars: - file: infra.yml - name: __infra - -- name: Combine with CDP Public Cloud environment runtime variables - ansible.builtin.set_fact: - globals: "{{ __infra.globals | default({}) | combine(globals | default({}), recursive=True) }}" - infra: "{{ __infra.infra | default({}) | combine(infra | default({}), recursive=True) }}" - env: "{{ __infra.env | default({}) | combine(env | default({}), recursive=True) }}" - -- name: Validate CDP Public Cloud environment variables - ansible.builtin.import_role: - name: cloudera.exe.infrastructure - tasks_from: validate.yml - environment: "{{ env_vars }}" - -- name: Initialize CDP Public Cloud environment teardown variables - ansible.builtin.import_role: - name: cloudera.exe.infrastructure - tasks_from: initialize_teardown.yml - environment: "{{ env_vars }}" - -- name: Teardown CDP Public Cloud environment instructure - ansible.builtin.import_role: - name: cloudera.exe.infrastructure - tasks_from: teardown.yml - environment: "{{ env_vars }}" diff --git a/tests/integration/targets/teardown_infra/vars/infra.yml b/tests/integration/targets/teardown_infra/vars/infra.yml deleted file mode 100644 index 13c471db..00000000 --- a/tests/integration/targets/teardown_infra/vars/infra.yml +++ /dev/null @@ -1,26 +0,0 @@ ---- - -# Copyright 2025 Cloudera, Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Default flags for teardown - -env: - tunnel: true # Delete any private networking elements - -infra: - teardown: - delete_network: true - delete_data: true - delete_ssh_key: true diff --git a/tests/integration/test_molecule_integration.py b/tests/integration/test_molecule_integration.py index 972485ff..32c9ced6 100644 --- a/tests/integration/test_molecule_integration.py +++ b/tests/integration/test_molecule_integration.py @@ -14,13 +14,15 @@ from __future__ import absolute_import, division, print_function -from pytest_ansible.molecule import MoleculeScenario +# TODO Reenable pytest Molecule testing +# from pytest_ansible.molecule import MoleculeScenario -def test_integration(molecule_scenario: MoleculeScenario) -> None: - """Run molecule for each scenario. - :param molecule_scenario: The molecule scenario object - """ - proc = molecule_scenario.test() - assert proc.returncode == 0 +# def test_integration(molecule_scenario: MoleculeScenario) -> None: +# """Run molecule for each scenario. + +# :param molecule_scenario: The molecule scenario object +# """ +# proc = molecule_scenario.test() +# assert proc.returncode == 0 From c8ddba2ee72a9dd4b7b01347f2ce49df6e5db7c7 Mon Sep 17 00:00:00 2001 From: Webster Mudge Date: Tue, 26 Aug 2025 12:07:16 -0400 Subject: [PATCH 2/3] Update imports for testing dependencies Signed-off-by: Webster Mudge --- tests/unit/conftest.py | 2 +- tests/unit/plugins/modules/jdk_facts/test_jdk_facts.py | 6 +----- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/tests/unit/conftest.py b/tests/unit/conftest.py index 5f710ab0..a8a77e4c 100644 --- a/tests/unit/conftest.py +++ b/tests/unit/conftest.py @@ -26,7 +26,7 @@ from ansible.module_utils import basic from ansible.module_utils.common.text.converters import to_bytes -from tests.unit import ( +from ansible_collections.cloudera.exe.tests.unit import ( AnsibleFailJson, AnsibleExitJson, ) diff --git a/tests/unit/plugins/modules/jdk_facts/test_jdk_facts.py b/tests/unit/plugins/modules/jdk_facts/test_jdk_facts.py index 06077dbb..37ef6d31 100644 --- a/tests/unit/plugins/modules/jdk_facts/test_jdk_facts.py +++ b/tests/unit/plugins/modules/jdk_facts/test_jdk_facts.py @@ -20,13 +20,9 @@ import pytest - -from tests.unit import ( - AnsibleExitJson, -) - from ansible.module_utils.basic import AnsibleModule +from ansible_collections.cloudera.exe.tests.unit import AnsibleExitJson from ansible_collections.cloudera.exe.plugins.modules import jdk_facts From f9462be641ce38cf316d0e05096de19efc1b18bf Mon Sep 17 00:00:00 2001 From: Webster Mudge Date: Tue, 26 Aug 2025 12:07:28 -0400 Subject: [PATCH 3/3] Update TESTING documentation Signed-off-by: Webster Mudge --- TESTING.md | 245 ++++++++++++++++++++++++----------------------------- 1 file changed, 111 insertions(+), 134 deletions(-) diff --git a/TESTING.md b/TESTING.md index 36b25ccf..b4f70d51 100644 --- a/TESTING.md +++ b/TESTING.md @@ -1,18 +1,22 @@ # Testing cloudera.exe -The collection is migrating from `ansible-test` to `pytest` and `molecule` for unit and integration testing. In addition, we use `pre-commit` to handle linting and formatting both for `git` commits and for various Github Action workflows in the repository. - -## Setup +This project uses [Hatch](https://hatch.pypa.io/dev/) to manage the project dependencies, testing environments, common utilities and scripts, and other activities. It also makes heavy use of [pytest](https://pytest.org), [Ansible Molecule](https://ansible.readthedocs.io/projects/molecule/), and employs the [EC2 driver](https://github.com/ansible-community/molecule-plugins) to test using cloud instances (if needed). We also use [pre-commit](https://pre-commit.com/), [antsibull-docs](https://ansible.readthedocs.io/projects/antsibull-docs/), and [ansible-lint](https://ansible.readthedocs.io/projects/lint/) for linting and hygiene. To set up a development and test environment for the collection, you need to: +1. Set up the Hatch build system 1. Set up the Ansible Collection and Role paths -1. Install Ansible and the Python dependencies -1. Install the collection and its dependencies 1. Configure the PYTHONPATH to use the correct location of the collections code -1. Install the Molecule driver dependencies +1. Install the Molecule driver dependencies (AWS resources) + +## Hatch Build System + +You should install `hatch` as [per its documentation](https://hatch.pypa.io/dev/install/#installers). `hatch` should be able to handle all dependencies, including Python versions and virtual environments. + +> [!danger] OSX `dirs.data` default! +> The [default data directory](https://hatch.pypa.io/1.13/config/hatch/#data) for `hatch` is `~/Library/Application Support/hatch`, which causes trouble for `molecule`! You might need to change this location to a path with spaces! -### Ansible Collection and Role Paths +## Ansible Collection and Role Paths You have to install your Ansible collections, both the collection under test and its dependencies, into the `ansible_collections//` folder structure. For the collection under test, run the following _in the parent directory of your choosing_: @@ -20,7 +24,7 @@ You have to install your Ansible collections, both the collection under test and git clone https://github.com/cloudera-labs/cloudera.exe.git ansible_collections/cloudera/exe ``` -Then create the `roles` directory in the _parent directory_: +Then create the `roles` directory in this _parent directory_: ```bash mkdir roles @@ -33,189 +37,175 @@ export ANSIBLE_COLLECTIONS_PATH=$(pwd) export ANSIBLE_ROLES_PATH="$(pwd)/roles" ``` -### Set the PYTHONPATH +## PYTHONPATH -Include the `ANSIBLE_COLLECTIONS_PATH` variable to the `PYTHONPATH` to allow module imports. +Make sure to include the `ANSIBLE_COLLECTIONS_PATH` variable to the `PYTHONPATH` to allow module imports. ```bash export PYTHONPATH="${ANSIBLE_COLLECTIONS_PATH}":"${PYTHONPATH}" ``` -### Ansible +# Linting and Commits + +The `pre-commit` Python application is used to manage linting and other housekeeping functions. The application is installed as a `git` hook and as a Github workflow check. -Set up a development `virtualenv` and install `ansible-core~=2.16.0` and `ansible-navigator`. +Commits and pull requests will fail if your contributions do not pass the `pre-commit` checks. You can check your work-in-progress code by running the following: ```bash -pip install ansible-core~=2.16.0 ansible-navigator +hatch run lint ``` -> [!warning] -> Installing `>=2.17` will require that the target hosts run Python 3.7. This requirement extends to RHEL 8.x and its `platform-python`, which means that `2.17` will not work on these platforms. - -### Python Dependencies - -Install the development and collection Python requirements from the `requirements-dev.txt` and `requirements.txt` files respectively in the project root. +Or manually: ```bash -pip install -r ansible_collections/cloudera/exe/requirements.txt -pip install -r ansible_collections/cloudera/exe/requirements-dev.txt +pre-commit run -a ``` -### Collection Dependencies +# pytest Testing -You also need to install the collection's dependencies and install them into the `ANSIBLE_COLLECTIONS_PATH`: +To see what tests (unit and integration) are available, run the following from the `ANSIBLE_COLLECTIONS_PATH` directory: ```bash -ansible-galaxy collection install -r ansible_collections/cloudera/exe/requirements.yml -p "${ANSIBLE_COLLECTIONS_PATH}" +pushd ${ANSIBLE_COLLECTIONS_PATH}; +pytest ansible_collections/cloudera/exe --collect-only; +popd; ``` -And install any role dependencies as well into the `ANSIBLE_ROLES_PATH`: +You should see something like: -```bash -ansible-galaxy role install -r ansible_collections/cloudera/exe/requirements.yml -p "${ANSIBLE_ROLES_PATH}" +``` +platform darwin -- Python 3.12.8, pytest-8.4.1, pluggy-1.6.0 +rootdir: /Users/wmudge/Devel/collections/ansible_collections/cloudera/exe +configfile: pyproject.toml +collected 128 items + + + + + + + + + + + + + + + + + + + + + + + + +``` + +To run all of the tests: + +```bash +pushd ${ANSIBLE_COLLECTIONS_PATH}; +pytest ansible_collections/cloudera/exe; +popd; ``` -If the collection has any system requirements, run `bindep` on its requirements file: +To run a selected test, execute with a regex: ```bash -bindep -f ansible_collections/cloudera/exe/bindep.txt +pushd ${ANSIBLE_COLLECTIONS_PATH}; +pytest ansible_collections/cloudera/exe -k "test_filter_version" +popd; ``` -### Molecule - -Running the `molecule` tests requires `podman` as the container engine, so you will need to install that service on your test machine. Read more about [Podman](https://podman.io/) or [Podman Desktop](https://podman-desktop.io/). +# Molecule Testing -## Testing +Running the `molecule` tests requires AWS [credentials](#aws-credentials) and [test network](#network-setup), as all tests offload the VMs to AWS. You can either run standalone `molecule`, for roles and more advanced integration testing, or `pytest`. The latter is set up to run any and all tests, including `molecule` scenarios. -### Running standalone `molecule` tests +## Running standalone `molecule` tests + +Running `molecule` is straightforward. First, enable the `hatch` environment: + +```bash +hatch shell molecule +``` + +The `molecule` scenarios are located within each of the roles of the collection. To run a scenario, change to the role as your current working directory and then run `molecule`. See the [molecule documentation](https://ansible.readthedocs.io/projects/molecule/) for details. -Currently, `molecule` scenarios are located in the `extensions/molecule` directory of the collection. To run a scenario, change to `extensions` as your current working directory and then run `molecule`. For example: +```bash +pushd roles/cm_agent; +molecule test; +popd; +``` + +> [!info] +> In some cases, you might need to set your `ANSIBLE_ROLES_PATH` to the `roles` directory of this project. + +Some common `molecule` commands (again, check the [molecule documentation](https://ansible.readthedocs.io/projects/molecule/) for full details): | Command | Description | | --- | --- | -| `molecule test -s rdbms_server_postgresql_14_tls` | Execute the full test lifecyle for the PostgreSQL 14 server role with TLS | -| `molecule create -s rdbms_server_postgresql_14_tls` | Create the `platforms`, i.e. the inventory, that are the target hosts of the role testing | -| `molecule prepare -s rdbms_server_postgresql_14_tls` | Prep the target hosts for testing the roles | -| `molecule converge -s rdbms_server_postgresql_14_tls` | Run the testing playbook, i.e. converge the test code, on the target hosts | -| `molecule side-effect -s rdbms_server_postgresql_14_tls` | Prep the target hosts, post-`converge`, for any additional setup prior to verification or idempotency testing | -| `molecule verify -s rdbms_server_postgresql_14_tls` | Verify the target hosts | -| `molecule cleanup -s rdbms_server_postgresql_14_tls` | Clean up any resources, for example, temporary files created on the controller | -| `molecule destroy -s rdbms_server_postgresql_14_tls` | Destroy the `platform` hosts | +| `molecule test` | Execute the full test lifecyle | +| `molecule create` | Create the `platforms`, i.e. the inventory, that are the target hosts of the role testing | +| `molecule prepare` | Prep the target hosts for testing the roles | +| `molecule converge` | Run the testing playbook, i.e. converge the test code, on the target hosts | +| `molecule side-effect` | Prep the target hosts, post-`converge`, for any additional setup prior to verification or idempotency testing | +| `molecule verify` | Verify the target hosts | +| `molecule cleanup` | Clean up any resources, for example, temporary files created on the controller | +| `molecule destroy` | Destroy the `platform` hosts | You can limit testing to a `platform` within a scenario by using the `-p/--platform-name` parameter (or via the `MOLECULE_PLATFORM_NAME` environment variable): ```bash -molecule test -s rdbms_server_postgresql_14_tls -p rhel9.4 +molecule test -p rhel9.4 ``` To stop tests from destroying the platforms after encountering an error (or at all, even on a successful test), pass the `--destroy=never` parameter: ```bash -molecule test -s rdbms_server_postgresql_14_tls -p rhel9.4 --destroy=never +molecule test -p rhel9.4 --destroy=never ``` You can log into a running platform via the `login` subcommand and the `-h/--host` parameter: ```bash -molecule login -s rdbms_server_postgresql_14_tls -h rhel9.4 +molecule login -h rhel9.4 ``` As well as pass extra parameters to the underlying playbook (`converge` command only!): ```bash -molecule converge -s rdbms_server_postgresql_14_tls -- -vvv -t tls_config +molecule converge -- -vvv -t tls_config ``` -### Running `pytest` tests +## Running `molecule` via `pytest` -We use the `ansible-pytest` plugin to run unit and integration tests for the collection. - -To see what tests (unit and integration) are available, run the following from the `ANSIBLE_COLLECTIONS_PATH` directory: - -```bash -pytest ansible_collections/cloudera/exe --collect-only -``` - -You should see something like: - -``` -platform darwin -- Python 3.12.4, pytest-8.3.3, pluggy-1.5.0 -ansible: 2.16.11 -rootdir: /Users/wmudge/Devel/ansible_collections/cloudera/exe -configfile: pyproject.toml -testpaths: tests -plugins: ansible-24.9.0, xdist-3.6.1 -collected 8 items - - - - - - - - - - - - - - - - - - -``` - -To run a selected test, execute with a regex: +> [!warning] +> Not yet implemented! -```bash -pytest ansible_collections/cloudera/exe -k "postgresql_14_tls" -``` +We use the `ansible-pytest` plugin to run unit and integration tests for the collection. To run a Molecule scenario on a selected platform, i.e. target host, set the platform via the environment variable: ```bash -MOLECULE_PLATFORM_NAME="rhel9.4" pytest ansible_collections/cloudera/exe -k "postgresql_14_tls" +MOLECULE_PLATFORM_NAME="rhel9.4" pytest ansible_collections/cloudera/exe -k "tktk" ``` > [!warning] > If you run `pytest` in the root of the collection, `pytest` will copies the current collection into the required Ansible collection path structure within the working directory. That is, running `pytest` at the root of the **collection** creates a `collection/ansible_collections//` **within** the collection. > Thus, our recommendation is that you can run `pytest` in at the root of the **Ansible collections path**. That is, run `pytest ansible_collections// ...` so that `pytest` doesn't have to bootstrap the collections path. -## Linting and Commits - -The `pre-commit` Python application is used to manage linting and other housekeeping functions. The application is installed as a `git` hook and as a Github workflow check. - -Commits and pull requests will fail if your contributions do not pass the `pre-commit` checks. You can check your work-in-progress code by running the following: - -```bash -pre-commit run -a -``` - ---- - -# Testing - -This project uses [Hatch](https://hatch.pypa.io/dev/) to manage the project's dependencies, testing environments, and other activities. It also makes heavy use of [Ansible Molecule](https://ansible.readthedocs.io/projects/molecule/) and employs the [EC2 driver](https://github.com/ansible-community/molecule-plugins) to test using cloud instances. - -## Setup - -### Hatch Build System - -You should install `hatch` as [per its documentation](https://hatch.pypa.io/dev/install/#installers). `hatch` should be able to handle all dependencies, including Python versions and virtual environments. - -> [!danger] OSX `dirs.data` default! -> The [default data directory](https://hatch.pypa.io/1.13/config/hatch/#data) for `hatch` is `~/Library/Application Support/hatch`, which causes trouble for `molecule`! You might need to change this location to a path with spaces! - -### AWS Credentials +## AWS Credentials You will need valid AWS credentials set in your environment for `molecule` to operate successfully. -In addition, make sure you have a default region declared, either in your `AWS_PROFILE` or `AWS_DEFAULT_REGION`. +In addition, make sure you have a default region declared, either in your `AWS_PROFILE` or `AWS_DEFAULT_REGION` environment variables. -### Test Network +## Network Setup `molecule` only manages the EC2 instances, not the VPC and associated networking. To set up the required infrastructure, you can run the following `hatch` commands. @@ -238,24 +228,11 @@ $(hatch run molecule:export-subnet) $(hatch run molecule:export-security-group) ``` -## Execution - -Running `molecule` is straightforward. First, enable the `hatch` environment: - -```bash -hatch shell molecule -``` - -And then change into the role directory that you wish to test, e.g. `cd roles/prereq_thp`, and run `molecule` from there. - -> [!info] -> In some cases, you might need to set your `ANSIBLE_ROLES_PATH` to the `roles` directory of this project. - -## Teardown +## Network Teardown First, make sure that you have destroyed any `molecule` scenarios that are using your test infrastructure. -And then, from the _project root_, run: +And then run: ```bash hatch run molecule:teardown