Restructured docs for AWS Lambda (elastic#158)

AlexanderWert · estolfo · bmorelli25 · bmorelli25 · commit cd71c2650459 · 2022-04-01T12:04:28.000-07:00
* Restructured docs for AWS Lambda

* Update docs/monitoring-aws-lambda.asciidoc

Co-authored-by: Emily S &lt;emily.s@elastic.co&gt;

* Update docs/monitoring-aws-lambda.asciidoc

Co-authored-by: Emily S &lt;emily.s@elastic.co&gt;

* Update docs/monitoring-aws-lambda.asciidoc

Co-authored-by: Emily S &lt;emily.s@elastic.co&gt;

* Update docs/monitoring-aws-lambda.asciidoc

Co-authored-by: Emily S &lt;emily.s@elastic.co&gt;

* add temp anchor to fix build

* Update docs/monitoring-aws-lambda.asciidoc

Co-authored-by: Brandon Morelli &lt;bmorelli25@gmail.com&gt;

* Update docs/monitoring-aws-lambda.asciidoc

Co-authored-by: Brandon Morelli &lt;bmorelli25@gmail.com&gt;

* Update docs/monitoring-aws-lambda.asciidoc

Co-authored-by: Brandon Morelli &lt;bmorelli25@gmail.com&gt;

* Update add-extension-layer.asciidoc

* Fixed review comments

* Adding missing config options

* docs: add env imgs

* add images

* Added Node.js env-var image

Co-authored-by: Emily S &lt;emily.s@elastic.co&gt;
Co-authored-by: bmorelli25 &lt;brandon.morelli@elastic.co&gt;
Co-authored-by: Brandon Morelli &lt;bmorelli25@gmail.com&gt;
Co-authored-by: Jean-Louis Voiseux &lt;48380853+jlvoiseux@users.noreply.github.com&gt;
# Conflicts:
#	docs/add-extension/add-extension-layer.asciidoc
#	docs/aws-lambda-extension.asciidoc
#	docs/lambda-selector/lambda-attributes-selector.asciidoc
#	docs/monitoring-aws-lambda.asciidoc
diff --git a/docs/add-extension/add-extension-layer.asciidoc b/docs/add-extension/add-extension-layer.asciidoc
@@ -5,7 +5,7 @@ To add a layer to a Lambda function through the AWS Management Console:
 1. Navigate to your function in the AWS Management Console
 2. Scroll to the Layers section and click the _Add a layer_ button image:images/config-layer.png[image of layer configuration section in AWS Console]
 3. Choose the _Specify an ARN_ radio button
-4. Copy and paste the following ARN of the APM Lambda Extension layer in the _Specify an ARN_ text input: + 
+4. Copy and paste the following ARN of the APM Lambda Extension layer in the _Specify an ARN_ text input: +
 +++<span style="font-size:10pt"><b>EXTENSION_ARN</b></span>+++
 image:images/choose-a-layer.png[image of choosing a layer in AWS Console]
 5. Click the _Add_ button
@@ -19,7 +19,7 @@ To add the layers to your Lambda function through the AWS Management Console:
 1. Navigate to your function in the AWS Management Console
 2. Scroll to the Layers section and click the _Add a layer_ button image:images/config-layer.png[image of layer configuration section in AWS Console]
 3. Choose the _Specify an ARN_ radio button
-4. Copy and paste the following ARNs of the APM Lambda Extension layer and the APM agent layer in the _Specify an ARN_ text input: + 
+4. Copy and paste the following ARNs of the APM Lambda Extension layer and the APM agent layer in the _Specify an ARN_ text input: +
 APM Extension layer: +
 +++<span style="font-size:10pt"><b>EXTENSION_ARN</b></span>+++ +
 APM Agent layer: +
@@ -61,12 +61,13 @@ In your SAM `template.yml` file add the APM Extension Layer ARN as follows:
 [source,yml]
 ----
 ...
-ServerlessFunction:
-  Type: AWS::Serverless::Function
-  Properties:
-    ...
-    Layers:
-        - EXTENSION_ARN
+Resources:
+  yourLambdaFunction:
+    Type: AWS::Serverless::Function
+    Properties:
+      ...
+      Layers:
+          - EXTENSION_ARN
 ...
 ----
 
@@ -79,13 +80,14 @@ In your SAM `template.yml` file add the Layer ARNs of the APM Extension and the
 [source,yml]
 ----
 ...
-ServerlessFunction:
-  Type: AWS::Serverless::Function
-  Properties:
-    ...
-    Layers:
-        - EXTENSION_ARN
-        - AGENT_ARN
+Resources:
+  yourLambdaFunction:
+    Type: AWS::Serverless::Function
+    Properties:
+      ...
+      Layers:
+          - EXTENSION_ARN
+          - AGENT_ARN
 ...
 ----
 
@@ -133,7 +135,6 @@ To add the APM Extension Layer to your function add the ARN to the `layers` prop
 ----
 ...
 resource "aws_lambda_function" "your_lambda_function" {
-  function_name = "yourLambdaFunctionName"
   ...
   layers = ["EXTENSION_ARN"]
 }
@@ -149,7 +150,6 @@ To add the APM Extension and the APM Agent to your function add the ARNs to the
 ----
 ...
 resource "aws_lambda_function" "your_lambda_function" {
-  function_name = "yourLambdaFunctionName"
   ...
   layers = ["EXTENSION_ARN", "AGENT_ARN"]
 }
diff --git a/docs/lambda-selector/lambda-attributes-selector.asciidoc b/docs/lambda-selector/lambda-attributes-selector.asciidoc
@@ -68,9 +68,9 @@ const addArnGenerator = async (type, ghRepo, arnPattern) => {
     const releases = await fetch(`https://api.github.com/repos/elastic/${ghRepo}/releases`).then(data => {
         return data.json();
       });
-      
+
       releases.filter(release => !release.draft && release.body.includes('arn:aws:lambda:'));
-      
+
       var latestRelease = releases[0];
 
       releases.forEach(release => {
@@ -107,7 +107,7 @@ window.addEventListener("DOMContentLoaded", async () => {
 </script>
 
 <div role="lambda-selector">
-  <div role="lambda-selector-header">Select the AWS region and architecture of your Lambda function:</div>
+  <div role="lambda-selector-header">Select the AWS region and architecture of your Lambda function. This documentation will update based on your selections.</div>
   <div role="lambda-selector-content">
     <div role="lambda-selector-input">
       <div>region:</div>
diff --git a/docs/monitoring-aws-lambda.asciidoc b/docs/monitoring-aws-lambda.asciidoc
@@ -1,194 +1,79 @@
-[[aws-lambda-extension]]
-= AWS Lambda Extension (Experimental)
-
-experimental::[]
-
-Elastic's APM Agents instrument AWS Lambda functions and dispatch APM data via an AWS Lambda Extension.
-
-[discrete]
-[[aws-lambda-arch-old]]
-== Extension Architecture
-
-Normally, during the execution of a Lambda function, there's only a single language process running in the AWS Lambda execution environment.  However, with an AWS Lambda Extension, Lambda users can run a _second_ process alongside their main service/application process.
-
-image:images/data-flow.png[image showing data flow from lambda function, to extension, to APM Server]
-
-By using a custom-built AWS Lambda Extension, Elastic APM Agents can send data to a locally running Lambda Extension process, and that process will forward data on to APM Server.  The Lambda Extension ensures that any latency between the Lambda function and the AWS Server instance will not cause latency in the Lambda function/Service itself.
-
-[discrete]
-[[aws-lambda-instrumenting]]
-== Instrumenting a Lambda Function
-
-The rest of this guide contains instructions for instrumenting a Lambda function. There are two high level steps to instrumenting an AWS Lambda function.
-
-1. <<aws-lambda-configure-layer>>
-2. <<aws-lambda-handler>>
-
-[discrete]
-[[aws-lambda-configure-layer]]
-=== Configuring the APM Lambda Extension Layer
-
-First, you'll need to https://github.com/elastic/apm-aws-lambda/releases[pick the right ARN from the extension release table] for your AWS Lambda function. The ARN has the pattern `arn:aws:lambda:<AWS_REGION_KEY>:267093732750:layer:elastic-apm-extension-<APM_EXTENSION_VERSION>-<ARCHITECTURE_KEY>:<LAYER_VERSION>` and depends on:
-
-* The AWS region your Lambda function runs in. The APM Lambda Extension layer needs to be in the same region as your Lambda function.
-* The architecture (_x86_64_ or _arm64_) used for your Lambda function.
-* The version of the APM Lambda Extension you would like to use.
+[[monitoring-aws-lambda]]
+= Monitoring AWS Lambda Functions
 
-You'll then need to configure your function to use that layer. To add a layer
+Elastic APM lets you monitor your AWS Lambda functions.
+The natural integration of {apm-guide-ref}/apm-distributed-tracing.html[distributed tracing] into your AWS Lambda functions provides insights into the functions' execution and runtime behavior as well as their relationships and dependencies to other services.
 
-1. Navigate to your function in the AWS Console
-2. Scroll to the Layers section and click the _Add a layer_ button image:images/config-layer.png[image of layer configuration section in AWS Console]
-3. Choose the _Specify an ARN_ radio button
-4. Enter the Version ARN of the APM Lambda Extension layer in the _Specify an ARN_ text input
-5. Click the _Add_ button
+To get started with the setup of Elastic APM for your Lambda functions, checkout the language-specific guides:
 
-[discrete]
-[[aws-lambda-env-vars]]
-==== Configure Environment Variables
+* {apm-node-ref}/lambda.html[Quick Start with APM on AWS Lambda - Node.js]
+* {apm-py-ref}/lambda-support.html[Quick Start with APM on AWS Lambda - Python]
+* {apm-java-ref}/aws-lambda.html[Quick Start with APM on AWS Lambda - Java]
 
-Finally, once the layer's in place you'll need to configure a few environment variables. To configure variables
+Learn more about the <<aws-lambda-arch, architecture>> of Elastic APM for AWS Lambda.
 
-1. Navigate to your function in the AWS Console
-2. Click on the _Configuration_ tab
-3. Click on _Environment variables_
-4. Add the necessary variables.
+[[aws-lambda-arch]]
+== APM Architecture for AWS Lambda
 
-The following environment variables are relevant for instrumenting a Lambda function:
+AWS Lambda uses a special execution model to provide a scalable, on-demand compute service for code execution. In particular, AWS freezes the execution environment of a lambda function when no active requests are being processed. This execution model poses additional requirements on APM in the context of AWS Lambda functions:
 
-* (required) `ELASTIC_APM_LAMBDA_APM_SERVER`: +
-This required config option controls where the Lambda extension will ship data. This should be the URL of the final APM Server destination for your telemetry.
-
-* (required) `ELASTIC_APM_SECRET_TOKEN` or `ELASTIC_APM_API_KEY`: +
-One of these needs to be set as the authentication method that the extension uses when sending data to the URL configured via `ELASTIC_APM_LAMBDA_APM_SERVER`.
-
-* (optional) `ELASTIC_APM_SERVICE_NAME`: +
-The configured name of your application or service.  The APM Agent will use this value when reporting data to APM Server. If unset, the APM Agent will automatically set the value based on the Lambda function name. Use this config option if you want to group multiple Lambda functions under a single service entity in APM.
+1. To avoid data loss, APM data collected by APM agents needs to be flushed before the execution environment of a lambda function is frozen.
+2. Flushing APM data must be fast so as not to impact the response times of lambda function requests.
 
-* (optional) `ELASTIC_APM_DATA_RECEIVER_TIMEOUT_SECONDS`: +
-The timeout value, in seconds, for the Lambda Extension's server receiving data from the agent. The _default_ is `15`.
-
-* (optional) `ELASTIC_APM_SEND_STRATEGY`: +
-Whether to synchronously flush APM agent data from the extension to the APM server at the end of the function invocation.
-The two accepted values are `background` and `syncflush`. The _default_ is `syncflush`.
-** The `background` strategy indicates that the extension will not flush when it receives a signal that the function invocation
-has completed. It will instead send any remaining buffered data on the next function invocation. The result is that, if the
-function is not subsequently invoked for that Lambda environment, the buffered data will be lost. However, for lambda functions
-that have a steadily frequent load pattern the extension could delay sending the data to the APM server to the next lambda
-request and do the sending in parallel to the processing of that next request. This potentially would improve both the lambda
-function response time and its throughput.
-** The other value, `syncflush` will synchronously flush all remaining buffered APM agent data to the APM server when the
-extension receives a signal that the function invocation has completed. This strategy blocks the lambda function from receiving
-the next request until the extension has flushed all the data. This has a negative effect on the throughput of the function,
-though it ensures that all APM data is sent to the APM server.
+To accomplish the above, Elastic's APM Agents instrument AWS Lambda functions and dispatch APM data via an https://docs.aws.amazon.com/lambda/latest/dg/using-extensions.html[AWS Lambda Extension].
 
-An example configuration of the APM Lambda Extension might look like the following
+Normally, during the execution of a Lambda function, there's only a single language process running in the AWS Lambda execution environment.  With an AWS Lambda Extension, Lambda users run a _second_ process alongside their main service/application process.
 
-[source,bash]
-----
-ELASTIC_APM_LAMBDA_APM_SERVER=https://your-apm-server-url:443 # (required) this is your APM Server URL
-ELASTIC_APM_SECRET_TOKEN=shhhhhhitsasecret     # (required) this is your APM Server's secret token
-ELASTIC_APM_SERVICE_NAME=yourApmServiceName    # (optional) use this to group multiple lambda functions
-ELASTIC_APM_DATA_RECEIVER_TIMEOUT_SECONDS=20   # (optional) wait 20s to receive data from the APM agent
-ELASTIC_APM_SEND_STRATEGY=background           # (optional) this is the default strategy
-----
+image:images/architecture-white.png[image showing data flow from lambda function, to extension, to APM Server]
 
-For a _minimal configuration_ you need to specify the `ELASTIC_APM_LAMBDA_APM_SERVER` and the `ELASTIC_APM_SECRET_TOKEN` config options.
-
-[discrete]
-[[aws-lambda-handler]]
-=== Configuring the Agent and Lambda Function handler
-
-After the extension has been installed, install the relevant language agent and wrap the Lambda function handler.
-
-
-[discrete]
-[[aws-lambda-nodejs]]
-==== Node.js
-
-To install the Node.js agent in Lambda you'll use a second layer.  You'll need to https://github.com/elastic/apm-agent-nodejs/releases[pick the right ARN from the agent release table]. The ARN has the pattern `arn:aws:lambda:<AWS_REGION_KEY>:267093732750:layer:elastic-apm-node-ver-<APM_EXTENSION_VERSION>:<LAYER_VERSION>` and depends on:
-
-* The AWS region your Lambda function runs in. The APM Lambda Extension layer needs to be in the same region as your Lambda function.
-* The version of the APM Lambda Extension you would like to use.
-
-You'll then need to configure your function to use that layer. To add a layer
-
-1. Navigate to your function in the AWS Console
-2. Scroll to the Layers section and click the _Add a layer_ button image:images/config-layer.png[image of layer configuration section in AWS Console]
-3. Choose the _Specify an ARN_ radio button
-4. Enter the Version ARN of the APM Lambda Extension layer in the _Specify an ARN_ text input
-5. Click the _Add_ button
-
-Finally, to have the Node.js agent automatically wrap your Lambda function handler, add the following environment variable.
-
-[source]
----
-NODE_OPTIONS=-r elastic-apm-node/start
----
-
-See the {apm-node-ref}/lambda.html[Node.js agent setup guide] for detailed instructions on setting up the Node.js agent for AWS Lambda.
-
-[discrete]
-[[aws-lambda-python]]
-==== Python
-
-In Python, you wrap a Lambda function handler using the following syntax.
-
-[source,python]
-----
-from elasticapm import capture_serverless
-@capture_serverless()
-def handler(event, context):
-    return {"statusCode": r.status_code, "body": "Success!"}
-----
-
-See the {apm-py-ref}/lambda-support.html[Python agent setup guide] for detailed instructions on setting up the Python agent for AWS Lambda.
-
-[discrete]
-[[aws-lambda-java]]
-==== Java
-
-Like the extension, the Elastic APM Java agent is installed as a Lambda layer.
-
-See the {apm-java-ref}/aws-lambda.html[Java agent setup guide] for detailed instructions on setting up the Java agent for AWS Lambda.
+By using an AWS Lambda Extension, Elastic APM Agents can send data to a local Lambda Extension process, and that process will forward data on to APM Server asynchronously. The Lambda Extension ensures that any potential latency between the Lambda function and the APM Server instance will not cause latency in the request flow of the Lambda function itself.
 
 [[aws-lambda-config-options]]
 == Configuration Options for APM on AWS Lambda
 
 The recommended way of configuring the APM Lambda Extension and the APM Agents on AWS Lambda is through the Lambda function's environment variables.
 
-[float]
+The configuration options for the APM Agents are documented in the corresponding language agents:
+
+* {apm-node-ref}/configuration.html[Configuration options - Node.js APM Agent]
+* {apm-py-ref}/configuration.html[Configuration options - Python APM Agent]
+* {apm-java-ref}/configuration.html[Configuration options - Java APM Agent]
+
+The following configuration options are particularly relevant for Elastic's APM on AWS Lambda:
+
+[[aws-lambda-extension]]
 === `ELASTIC_APM_LAMBDA_APM_SERVER`
 This required config option controls where the Lambda extension will ship data. This should be the URL of the final APM Server destination for your telemetry.
 
-[float]
 === `ELASTIC_APM_SECRET_TOKEN` or `ELASTIC_APM_API_KEY`
 One of these needs to be set as the authentication method that the extension uses when sending data to the URL configured via `ELASTIC_APM_LAMBDA_APM_SERVER`.
 
-[float]
 === `ELASTIC_APM_SERVICE_NAME`
 The configured name of your application or service.  The APM Agent will use this value when reporting data to the APM Server. If unset, the APM Agent will automatically set the value based on the Lambda function name. Use this config option if you want to group multiple Lambda functions under a single service entity in APM.
 
-[float]
 === `ELASTIC_APM_DATA_RECEIVER_TIMEOUT_SECONDS`
-The timeout value, in seconds, for the Lambda Extension's server receiving data from the agent. The _default_ is `15`.
+The APM Lambda Extension's timeout value, in seconds, for receiving data from the APM Agent. The _default_ is `15`.
+
+=== `ELASTIC_APM_DATA_RECEIVER_SERVER_PORT`
+The port on which the APM Lambda Extension listens to receive data from the APM Agent. The _default_ is `8200`.
+
+=== `ELASTIC_APM_DATA_FORWARDER_TIMEOUT_SECONDS`
+The timeout value, in seconds, for the Lambda Extension's HTTP client sending data to the APM Server. The _default_ is `3`. If the Extension's attempt to send APM data during this time interval is not successful, the extension queues back the data. Further attempts at sending the data are governed by an exponential backoff algorithm: data will be sent after a increasingly large grace period of 0, then circa 1, 4, 9, 16, 25 and 36 seconds, provided that the Lambda function execution is ongoing.
 
-[float]
 === `ELASTIC_APM_SEND_STRATEGY`
-Whether to synchronously flush APM agent data from the extension to the APM server at the end of the function invocation.
+Whether to synchronously flush APM agent data from the extension to the APM Server at the end of the function invocation.
 The two accepted values are `background` and `syncflush`. The _default_ is `syncflush`.
 
 * The `background` strategy indicates that the extension will not flush when it receives a signal that the function invocation
 has completed. It will instead send any remaining buffered data on the next function invocation. The result is that, if the
 function is not subsequently invoked for that Lambda environment, the buffered data will be lost. However, for lambda functions
-that have a steadily frequent load pattern the extension could delay sending the data to the APM server to the next lambda
+that have a steadily frequent load pattern the extension could delay sending the data to the APM Server to the next lambda
 request and do the sending in parallel to the processing of that next request. This potentially would improve both the lambda
 function response time and its throughput.
-* The other value, `syncflush` will synchronously flush all remaining buffered APM agent data to the APM server when the
+* The other value, `syncflush` will synchronously flush all remaining buffered APM agent data to the APM Server when the
 extension receives a signal that the function invocation has completed. This strategy blocks the lambda function from receiving
 the next request until the extension has flushed all the data. This has a negative effect on the throughput of the function,
 though it ensures that all APM data is sent to the APM server.
 
-[role="exclude",id="aws-lambda-arch"]
-== AWS Lambda architecture
-
-Please see <<aws-lambda-arch>> instead. This link will be fixed soon.
+=== `ELASTIC_APM_LOG_LEVEL`
+The logging level to be used by both the APM Agent and the Lambda Extension. Supported values are `trace`, `debug`, `info`, `warning`, `error`, `fatal` and `panic`.
