End-to-end testing using SAM CLI and locally run Lambdas

[jlvoiseux]

Motivation / Summary

The work leading to this PR was initiated based on PR #67, which is effectively replaced by the following Pull Request. The objective is to provide contributors with an automated method to test the APM data flow end-to-end, in the context of AWS lambda functions.

Files

This PR adds a subfolder e2e-testing to the extension folder. This subfolder contains a Go script e2e_test.go, which is the end-to-end test method itself. Alongside the Go script, an additional .gitignore three folders containing the lambda function to be built are provided. These three folders correspond to the 3 languages for which the APM agent currently supports the AWS Lambda extension : Javascript (Node), Python and Java.

Checklist

• Make the execution of the Lambda functions fully local
• Automate the retrieval and the permission changes on the APM Java agent layer
• Test APM-instrumented application -> Elasticsearch data flow through a UUID-named transaction
• Use the Golang testing framework instead of the main package
• By default, remove the E2E test from the default testing suite (an environment variable has to be set explicitely in order to run the test
• Add an additional check to confirm that the APM extension is running and healthy in the test containers
• Parallelise the tests for each of the 3 languages : 8e955a7
• Make the test usable with both a local Elasticsearch instance and a Cloud instance
• Use a mock APM server to remove reliance on Elasticsearch : cdffa11
• Do not create a new transaction, but change the name of existing ones (to verify the auto-instrumentation by the current agent) : 3457f80
• Make formatting compliant
• Close PR Use sam cli to test lambda function locally #67

How to test

• Install the SAM CLI. Creating an AWS account is actually not required.
• Download / Clone the repository
• In apm-lambda-extension/e2e-testing, create a .env file and set the following environment variable :

RUN_E2E_TESTS=true

• Other optional environment variables :

FORCE_REBUILD_LOCAL_SAM_LAMBDAS=false
PARALLEL_EXECUTION=true
DEBUG_OUTPUT=false
TIMEOUT_NODE=20
TIMEOUT_PYTHON=20
TIMEOUT_JAVA=20
APM_AGENT_JAVA_VERSION=1.28.4

Note that if you do not set RUN_E2E_TESTS, the test will be skipped.

• Make sure that the docker daemon is up and running
• Run the test

cd e2e-testing
go test

[ghost]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-02-04T15:34:08.274+0000

• Duration: 8 min 19 sec

Test stats 🧪

Test	Results
Failed	0
Passed	84
Skipped	6
Total	90

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[jlvoiseux]

Commit cdffa11 implements a mock APM Server, in order to make the testing module as standalone as possible. This impacts the testing strategy as follows :

• The mock server is initialized, and its url is passed to the method tasked with invoking the Lambda function. The server actually runs on localhost:serverPort, which can be accessed by the containers through http://host.docker.internal:serverPort.
• When the server receives a transaction from the APM Agent/extension, it decodes it and appends the document to a log file
• At the end of the test, the log is parsed to see if it contains the UUIDs passed in parameter to the Lambda functions.

[estolfo]

Thanks for your work on this!

[jlvoiseux]

The latest commit focuses on refactoring the execution of the test so that the execution is not blocked by a hanging test.

Changes

• Environment variables are not assigned a default value if missing
• Added three new environment variables : TIMEOUT_JAVA, TIMEOUT_NODE and TIMEOUT_PYTHON
• The parallel execution now stops when the maximum timeout is reached
• Sequential execution has been refactored to stop when the language-specific timeout is reached

[estolfo]

Just wanted to document here that we had a slack convo about simplifying the test script to only run one language, which would be specified via args. The test runner could parallelize as needed.

[estolfo]

Just a couple of questions and comments. It's almost complete 🎉

[estolfo]

🎉