ci: add smoke tests integration

[kruskall]

Initial work to integrate smoke testing scripts into CI.

Mostly inspired by the work in the mutating webhook and APM server, add make tasks to make testing easier and provide a target for CI to discover, run and cleanup the environment.

The tasks are flexible and support muiltiple smoke tests (currently there's only one).

Related to #217

[ghost]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-11-01T10:57:40.639+0000

• Duration: 8 min 59 sec

Test stats 🧪

Test	Results
Failed	0
Passed	176
Skipped	2
Total	178

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[simitt]

@kruskall could you provide a sentence or two to the description.

@v1v could you (or someone from the team) review the ci automation part here please (does not need to be merged before 8.5 FF).

[v1v]

A few comments, but would you mind splitting this PR in two, so the .ci/jobs/** are in a different PR, so we can easily merge and test whether the pipeline in this particular PR works as expected?

.ci/jobs are quite static, so jobs are not generated in the CI as long as they are not merged, while .ci/*.groovy are more dynamic as long as the Jenkins Multibranch Job has been created.

[kruskall]

Sorry about the delay, there were a few issues with the current script (e.g. #319).

I'll pick this up again today

[v1v]

LGTM, but let's wait for #329 so we can test this PR without merging it

[v1v]

I've just added the support to run always when pushing into the main branch by default. And also support for running on PR basis by using the Jenkins UI.

For instance, this particular PR can be validated manually by running the job -> https://apm-ci.elastic.co/job/library/job/apm-aws-lambda-smoketests-mbp/view/change-requests/job/PR-310/

[kruskall]

@v1v Thank you for that!

It seems the job is failing because the aws provider is not configured (missing credentials), could you help with that ? I'm not familiar with the process/guidelines regarding adding credentials to jenkins.

[v1v]

It seems the job is failing because the aws provider is not configured (missing credentials), could you help with that ? I'm not familiar with the process/guidelines regarding adding credentials to jenkins.

Everything is in Vault, as far as I know it uses the secrets defined in secret/observability-team/ci/elastic-observability-aws-account-auth. https://github.com/elastic/apm-aws-lambda/pull/310/files#diff-377731352d232f58c31c9af00e5e8dbbf25eb9a2db0edb045a9853ab9d99fd99R115 is the section to configure AWS accordingly with the command aws configure import csv file

What's the error? Maybe there are some missing permissions for the existing AWS account?

[kruskall]

What's the error? Maybe there are some missing permissions for the existing AWS account?

[2022-10-12T12:14:35.035Z] Error: error configuring Terraform AWS Provider: failed to get shared config profile, [email protected]
 
[2022-10-12T12:14:35.035Z] 
[2022-10-12T12:14:35.035Z]   with provider["registry.terraform.io/hashicorp/aws"],
[2022-10-12T12:14:35.035Z]   on main.tf line 1, in provider "aws":
[2022-10-12T12:14:35.035Z]    1: provider "aws" {

See https://apm-ci.elastic.co/blue/organizations/jenkins/library%2Fapm-aws-lambda-smoketests-mbp/detail/PR-310/5/pipeline

[v1v]

I'm not much familiar with that error, I just searched and found:

If I downgrade to version = "~> 3.0" it fixes the problem. But I would like to upgrade to the latest version.

In here

I also see some references about some specific terraform configuration:

• Terraform apply fails to configure AWS provider with shared credential file hashicorp/terraform-provider-aws#15361 (comment)
• https://registry.terraform.io/providers/hashicorp/aws/latest/docs#shared-configuration-and-credentials-files

I'm not much familiar with the existing terraform/aws configuration for this project, that particular withAWSEnv steps works in some other projects, do you happen to know what's the profile meaning in terms of terraform?