Maintenance: run e2e tests in parallel

[am29d]

Summary

While we run e2e tests in a matrix:

    strategy:
      matrix:
        package: [logger, metrics, tracer, parameters]
        version: [14, 16, 18]
      fail-fast: false

it would be great to also be able to execute these tests during development.

Why is this needed?

To reduce feedback cycle time for e2e tests during development.

Which area does this relate to?

Automation

Solution

@dreamorosi already opened a discussion, and the pointers to RFCs. I don't have a solution yet, so any recommendation and contribution is appreciated.

Acknowledgment

• This request meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[dreamorosi]

If you run them with npm run test:e2e -ws it will run them sequentially for one runtime (Node.js 18).

If we want to run them in parallel we could write a bash script that does that like npm run test:e2e -w packages/tracer & npm run test:e2e -w packages/logger & ....

This however will mix all the stdout & stderr making it hard to read what's happening in case of multiple failures. This is also what happens with lerna, which is able to run things in parallel and is already available in the repo (but used only for version and release).

This is something we should keep our eyes on, however in most cases I'm running locally only specific groups/utilities and then run the full suite only on the repo. This makes it also easier to collaborate on results and troubleshooting when needed.

[dreamorosi]

Been looking into this while working on other issues and running the tests in parallel with a fairly crude method caused me to hit rate limits on deployment.

Specifically, I ran three terminal sessions like this:

• session 1: RUNTIME=nodejs14x npm run test:e2e -w packages/tracer
• session 2: RUNTIME=nodejs16x npm run test:e2e -w packages/tracer
• session 3: npm run test:e2e -w packages/tracer

I wonder if this will be an issue if we try to run multiple packages / runtimes combos concurrently.