ci: automate ARN publishing #101

v1v · 2022-01-13T11:14:33Z

What

Publish the AWS lambda as an ARN for AWS users in various regions for linux and arm.

It happens when a new Git tag is created with the format v.major.minor.patch then:

Create the binaries for AMD64 and ARM64
Publish the AWS lambda in each AWS region
Create a release entry in the GitHub UI that includes the ARN for each region and platform and the list of changes.

Refactored the Makefile to avoid duplicated code and added section in the DEVELOPMENT.md regarding this new release automation.

Actions

Login to AWS. status: Waiting for the new AWS account.
List of Regions where to publish. status: Is it possible to list this dynamically?
Make publish to the above given regions and store the ARN to be consumed. status: to be implemented
Create Release Notes. status: to be implemented
Enable only for the tag releases. status: uncomment the code in the CI

Implementation details

Delegate all the logic for build/test/distribute and publish in AWS to the Makefile. This should help to run locally the same script, so as long as you have the right permissions then it should work out of the box.
CI should only provision workers, configure credentials, checkout the source code and run the make goals.

Docs

See the doc to list all the AWS regions.

Test

Pipeline

Release notes

An example of the release notes that are generated in my forked repo (unfortunately the autogenerated does not work in my end while working on a PR)

https://github.com/v1v/apm-aws-lambda/releases/tag/untagged-a2134f29781f369bef01

List all the AWS regions and publish (dry-run)

$ make -C apm-lambda-extension publish-in-all-aws-regions
Makefile:47: *** ELASTIC_LAYER_NAME is undefined.  Stop.
$ ELASTIC_LAYER_NAME=my-test make -C apm-lambda-extension publish-in-all-aws-regions
publish "my-test" in af-south-1
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in eu-north-1
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in ap-south-1
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in eu-west-3
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in eu-west-2
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in eu-south-1
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in eu-west-1
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in ap-northeast-3
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in ap-northeast-2
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in me-south-1
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in ap-northeast-1
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in sa-east-1
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in ca-central-1
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in ap-east-1
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in ap-southeast-1
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in ap-southeast-2
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in eu-central-1
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in ap-southeast-3
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in us-east-1
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in us-east-2
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in us-west-1
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"
publish "my-test" in us-west-2
aws lambda publish-layer-version --layer-name "my-test" --zip-file "fileb://./bin/extension.zip"

Local testing

AWS_SECRET_ACCESS_KEY="..." \
AWS_ACCESS_KEY_ID=... \
SUFFIX_ARN_FILE=arn.md \
ELASTIC_LAYER_NAME=v1v-aws-lambda-test \
make -C apm-lambda-extension publish-in-all-aws-regions

publish "v1v-aws-lambda-test" in eu-north-1
publish "v1v-aws-lambda-test" in ap-south-1
publish "v1v-aws-lambda-test" in eu-west-3
publish "v1v-aws-lambda-test" in eu-west-2
publish "v1v-aws-lambda-test" in eu-west-1
publish "v1v-aws-lambda-test" in ap-northeast-3
publish "v1v-aws-lambda-test" in ap-northeast-2
publish "v1v-aws-lambda-test" in ap-northeast-1
publish "v1v-aws-lambda-test" in sa-east-1
publish "v1v-aws-lambda-test" in ca-central-1
publish "v1v-aws-lambda-test" in ap-southeast-1
publish "v1v-aws-lambda-test" in ap-southeast-2
publish "v1v-aws-lambda-test" in eu-central-1
publish "v1v-aws-lambda-test" in us-east-1
publish "v1v-aws-lambda-test" in us-east-2
publish "v1v-aws-lambda-test" in us-west-1
publish "v1v-aws-lambda-test" in us-west-2
/Library/Developer/CommandLineTools/usr/bin/make create-arn-file

BRANCH_NAME=v0.0.99 \
SUFFIX_ARN_FILE=arn.md \
ELASTIC_LAYER_NAME=v1v-aws-lambda-test \
make -C apm-lambda-extension release-notes
v0.0.2 Prerelease  Pre-release  (v0.0.2)  about 1 month ago
v0.0.1 Prerelease  Pre-release  (v0.0.1)  about 3 months ago
cat *-arn.md > arn.md
https://github.com/v1v/apm-aws-lambda/releases/tag/untagged-adf8d18b440e71e6304a

ghost · 2022-01-13T11:28:27Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Reason: null
Start Time: 2022-01-27T20:06:35.409+0000
Duration: 11 min 52 sec
Commit: 1b1d08e

Test stats 🧪

Test	Results
Failed	0
Passed	84
Skipped	0
Total	84

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

v1v · 2022-01-13T15:02:04Z

/test

Prints a JSON skeleton to standard output without sending an API request

v1v · 2022-01-13T17:16:27Z

apm-lambda-extension/Makefile

+		publish-layer-version \
+		--layer-name "${ELASTIC_LAYER_NAME}" \
+		--zip-file "fileb://./bin/extension.zip" \
+		--generate-cli-skeleton > $(AWS_FOLDER)/${AWS_DEFAULT_REGION}


--generate-cli-skeleton will do nothing -> dry-run for now

v1v · 2022-01-13T17:17:17Z

.ci/Jenkinsfile

+                unstash "arn-arm"
+                unstash "arn-amd"


This is the trick to converge the outcome from the publishing stage for the two supported architectures (ARM and AMD64)

v1v · 2022-01-13T21:30:20Z

apm-lambda-extension/Makefile

-ifndef AWS_DEFAULT_REGION
-	$(error AWS_DEFAULT_REGION is undefined)
-endif
+build-and-publish: validate-layer-name validate-aws-default-region


validate-* are the goals to avoid duplication of missing arguments/env-variables in some other Make goals

v1v · 2022-01-13T21:30:28Z

apm-lambda-extension/Makefile

 test:
 	go test extension/*.go -v
 env:
-ifndef GOARCH


Already defined above

v1v · 2022-01-13T21:30:52Z

apm-lambda-extension/Makefile

+		release \
+		create $(BRANCH_NAME) \
+		--title '$(BRANCH_NAME)' \
+		--draft \


This is only for testing purposes

v1v · 2022-01-13T21:31:22Z

apm-lambda-extension/Makefile

+		create $(BRANCH_NAME) \
+		--title '$(BRANCH_NAME)' \
+		--draft \
+		--generate-notes \


Automatically generate title and notes for the release, it works in conjunction with --notes-file

This reverts commit f8d1173.

astorm

Things look good on the publishing a list of ARNs front, so +1 there (although the jenkins build syntax isn't something I'm as familiar with, so I can't comment in depth on that)

One problem I noticed is the changes to the Makefile appear to break the user-facing installer. If you look at this docs section you can see users have the self-service option of installing the extension by running a small Node.js command line program.

When running the program on the main branch, it successfully builds and installs the Lambda extension (as expected). However, when we attempt to run it with this PR branch, it fails half way through with the following error.

$ AWS_DEFAULT_REGION=... AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... ./elastic-lambda.js install
//...
# building extension and publishing as layer
stdout: running cd .. && make build-and-publish
error: Command failed: cd .. && make build-and-publish
Makefile:81: *** BRANCH_NAME is undefined.  Stop.
make: *** [build-and-publish] Error 2

I don't have any strong opinions on how we achieve this, but users being able to self serve by running this program is an important part of the this product.

Can we ensure that this functionality remains in place?

apm-lambda-extension/Makefile

A misleading error when running the publish - PublishLayerVersion operation: The security token included in the request is invalid.

astorm · 2022-01-27T17:44:32Z

@v1v Looks like we cleared up the previous errors here -- huzzah!

However, I'm now seeing a new error when running the installer -- specifically

% AWS_DEFAULT_REGION=... AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... ./elastic-lambda.js install

/* ... */

# building extension and publishing as layer
error: Command failed: /path/to/apm-aws-lambda/apm-lambda-extension/cli/elastic-lambda.js build-and-publish
undefined:1



SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at getLastJsonFromShellOutput (/path/to/apm-aws-lambda/apm-lambda-extension/cli/build-and-publish.js:37:23)
    at /path/to/apm-aws-lambda/apm-lambda-extension/cli/build-and-publish.js:57:22
    at ChildProcess.exithandler (child_process.js:299:7)
    at ChildProcess.emit (events.js:315:20)
    at maybeClose (internal/child_process.js:1048:16)
    at Socket.<anonymous> (internal/child_process.js:439:11)
    at Socket.emit (events.js:315:20)
    at Pipe.<anonymous> (net.js:673:12)

stderr: undefined:1

/* ... */

encountered error, exiting

% echo $?
1

I believe this is result of our changing the invocation of the publish-layer-version such that it no longer outputs to stdout, and instead outputs to a file

https://github.com/elastic/apm-aws-lambda/pull/101/files#diff-a402b77bda476fe59d07fb66cd01a490fb31ef84f423783e2bd7799a5db9596dR61-R68

The ./elastic-lambda.js install command processes the stdout results of publish-layer-version looking for a JSON string. With the new Makefile this string ends up in a file instead of stdout. The installer doesn't see the JSON, tries to process a blank string as a JSON string, and then exits with an error when it can't (since, without the information in this file, it can't proceed)

I'm unsure what the intent of this change is, but if I was trying to fix this I'd replace with the redirect (>) with a | into tee

	@aws lambda \
		--output json \
		publish-layer-version \
		--layer-name "${ELASTIC_LAYER_NAME}" \
		--description "AWS Lambda Extension Layer for Elastic APM" \
		--license "Apache-2.0" \
		--compatible-architectures "$(ARCHITECTURE)" \
		--zip-file "fileb://./bin/extension.zip"  | tee $(AWS_FOLDER)/${AWS_DEFAULT_REGION}

Assuming tee is available in our build environment, this should maintain the preexisting behavior of outputting to stdout (allowing the installer to read that JSON), while also writing the results out to the $(AWS_FOLDER)/${AWS_DEFAULT_REGION} file.

…h.js

v1v · 2022-01-27T20:12:46Z

Thanks @astorm for your review! 1b1d08e should now provide the compatibility with the existing process in addition to the new proposed here. No more breaking changes! Thanks again

v1v · 2022-01-27T20:14:50Z

apm-lambda-extension/Makefile

+	$(MAKE) create-arn-file
+
+# Publish the given LAYER in the given AWS region
+publish: validate-layer-name validate-aws-default-region


This goal is uses in two different places:

The traditional one when using ./elastic-lambda.js install

The one to rollout the ARN in all the AWS regions, see publish-in-all-aws-regions

I decided to refactor build-and-publish in favour of avoiding running aws lambda command in two different Make goals.

astorm

Thanks for making those changes @v1v -- ./elastic-lambda.js build-and-publish appears to be doing its thing again 👍

v1v added 3 commits January 13, 2022 10:44

ci: add release stage when a tag is created

8812080

ci: add release dist

fbbf425

chore: testing purposes

e27ca64

github-actions bot added the aws-λ-extension AWS Lambda Extension label Jan 13, 2022

v1v added 7 commits January 13, 2022 12:29

ci: stash is not needed when running a matrix

0c89783

ci: add release-notes

f5ed12e

chore: for testing purposes

8602116

ci: enable aws login

ffa2caa

ci: publish in all the AWS regions

7f3e9d0

chore: for testing purposes

019d6f0

ci: fix post location

4d39efe

v1v self-assigned this Jan 13, 2022

fix: You must specify a region.

620d8b0

v1v mentioned this pull request Jan 13, 2022

withAWSEnv: add support for ARM elastic/apm-pipeline-library#1499

Merged

makefile: report pipe errors

9ec1143

v1v added 2 commits January 13, 2022 16:17

Use generate-cli-skeleton for dry-run

3b9c99c

Prints a JSON skeleton to standard output without sending an API request

ci: create ARN table for each arch

97e47db

v1v commented Jan 13, 2022

View reviewed changes

v1v added 5 commits January 13, 2022 20:59

ci: remove duplicated stash

b597115

Support upload assets

1a43178

enable release notes with a draft

ccc7585

remove duplication

5bc534a

makefile: refactor validation in goals

4d37792

v1v commented Jan 13, 2022

View reviewed changes

apm-lambda-extension/Makefile

test:

go test extension/*.go -v

env:

ifndef GOARCH

Copy link

Member Author

v1v Jan 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already defined above

v1v commented Jan 13, 2022

View reviewed changes

v1v added 5 commits January 14, 2022 09:02

Revert "ci: GOARCH is not defined when publishing but needed"

323d959

This reverts commit f8d1173.

make: dist is the one in charge to prepare the arch binary

d993397

debug: for testing purposes

cb676e7

ci: create-are requires the goarch

fd7f857

debug: for testing purposes

5e563f4

v1v requested review from a team and astorm January 14, 2022 16:18

astorm suggested changes Jan 26, 2022

View reviewed changes

v1v commented Jan 26, 2022

View reviewed changes

apm-lambda-extension/Makefile Outdated Show resolved Hide resolved

v1v and others added 10 commits January 26, 2022 21:23

publish goal does not require branch validation

5deb100

Add license and description fields

19c9bc2

use the LayerVersionArn field

900d51f

Use a different arch

a8a5441

unrequired validation

669105b

use suffix since it's the correct value

b93bc14

enable when tag is created

63a925e

dry-run disabled

65a7124

no more debugging

f197135

aws: no all regions support the aws lambda publising

f5d4448

A misleading error when running the publish - PublishLayerVersion operation: The security token included in the request is invalid.

v1v marked this pull request as ready for review January 26, 2022 22:59

enable --compatible-architectures

05919ed

keep compatibility with the apm-lambda-extension/cli/build-and-publis…

1b1d08e

…h.js

v1v commented Jan 27, 2022

View reviewed changes

astorm approved these changes Feb 1, 2022

View reviewed changes

astorm mentioned this pull request Feb 1, 2022

Published ARN Names should Include Release Version and "Elastic" #110

Closed

2 tasks

estolfo merged commit aa6793f into elastic:main Feb 2, 2022

v1v mentioned this pull request Feb 2, 2022

[jjbb] enable tag releases in Jenkins #111

Merged

v1v deleted the feature/support-release-automation branch February 4, 2022 19:14

ci: automate ARN publishing #101

ci: automate ARN publishing #101

Uh oh!

Conversation

v1v commented Jan 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Actions

Implementation details

Docs

Test

Pipeline

Release notes

List all the AWS regions and publish (dry-run)

Local testing

Uh oh!

ghost commented Jan 13, 2022 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

v1v commented Jan 13, 2022

Uh oh!

v1v Jan 13, 2022

Choose a reason for hiding this comment

Uh oh!

v1v Jan 13, 2022

Choose a reason for hiding this comment

Uh oh!

v1v Jan 13, 2022

Choose a reason for hiding this comment

Uh oh!

v1v Jan 13, 2022

Choose a reason for hiding this comment

Uh oh!

v1v Jan 13, 2022

Choose a reason for hiding this comment

Uh oh!

v1v Jan 13, 2022

Choose a reason for hiding this comment

Uh oh!

astorm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

astorm commented Jan 27, 2022

Uh oh!

v1v commented Jan 27, 2022

Uh oh!

v1v Jan 27, 2022

Choose a reason for hiding this comment

Uh oh!

astorm left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

v1v commented Jan 13, 2022 •

edited

Loading

ghost commented Jan 13, 2022 •

edited by ghost

Loading