Added docs section for using secrets manager with the extension

docs/monitoring-aws-lambda.asciidoc

@@ -45,8 +45,15 @@ The following configuration options are particularly relevant for Elastic's APM
 === `ELASTIC_APM_LAMBDA_APM_SERVER`
 This required config option controls where the Lambda extension will ship data. This should be the URL of the final APM Server destination for your telemetry.
 
+[[aws-lambda-config-authentication-keys]]
 === `ELASTIC_APM_SECRET_TOKEN` or `ELASTIC_APM_API_KEY`
-One of these needs to be set as the authentication method that the extension uses when sending data to the URL configured via `ELASTIC_APM_LAMBDA_APM_SERVER`. Sending data to the APM Server if none of these options is set is possible, but your APM agent must be allowed to send data to your APM server in https://www.elastic.co/guide/en/apm/guide/current/configuration-anonymous.html[anonymous mode].
+One of these (or, alternatively, the corresponding settings for the AWS Secrets Manager IDs) needs to be set as the authentication method that the extension uses when sending data to the URL configured via `ELASTIC_APM_LAMBDA_APM_SERVER`. Alternatively, you can store your APM Server credentials <<aws-lambda-secrets-manager, using the AWS Secrets Manager>> and use the <<aws-lambda-config-secrets-manager-options>> config options, instead. Sending data to the APM Server if none of these options is set is possible, but your APM agent must be allowed to send data to your APM server in https://www.elastic.co/guide/en/apm/guide/current/configuration-anonymous.html[anonymous mode].
+
+[[aws-lambda-config-secrets-manager-options]]
+=== `ELASTIC_APM_SECRETS_MANAGER_SECRET_TOKEN_ID` or `ELASTIC_APM_SECRETS_MANAGER_API_KEY_ID`
+Instead of specifying the <<aws-lambda-config-authentication-keys>> as plain text in your Lambda environment variables, you can <<aws-lambda-secrets-manager, use the AWS Secrets Manager>> to securely store your APM authetication keys. The `ELASTIC_APM_SECRETS_MANAGER_API_KEY_ID` or `ELASTIC_APM_SECRETS_MANAGER_SECRET_TOKEN_ID` config options allow you to specify the Secrets Manager's secret id of the stored APM API key or APM secret token, respectively, to be used by the APM Lambda Extension for authentication.
+
+`ELASTIC_APM_SECRETS_MANAGER_SECRET_TOKEN_ID` takes precedence over <<aws-lambda-config-authentication-keys, `ELASTIC_APM_SECRET_TOKEN`>>, and `ELASTIC_APM_SECRETS_MANAGER_API_KEY_ID` over <<aws-lambda-config-authentication-keys, `ELASTIC_APM_API_KEY`>>, respectively.
 
 === `ELASTIC_APM_SERVICE_NAME`
 The configured name of your application or service.  The APM Agent will use this value when reporting data to the APM Server. If unset, the APM Agent will automatically set the value based on the Lambda function name. Use this config option if you want to group multiple Lambda functions under a single service entity in APM.
@@ -77,3 +84,41 @@ though it ensures that all APM data is sent to the APM server.
 
 === `ELASTIC_APM_LOG_LEVEL`
 The logging level to be used by both the APM Agent and the Lambda Extension. Supported values are `trace`, `debug`, `info`, `warning`, `error`, `critical` and `off`.
+
+[[aws-lambda-secrets-manager]]
+== Using AWS Secrets Manager to manage APM authentication keys
+When using the config options <<aws-lambda-config-authentication-keys>> for authentication of the APM Lambda Extension with Elastic APM, the corresponding keys are specified in plain text in the environment variables of your Lambda function. If you prefer to securely store the authentication keys, you can use the AWS Secrets Manager and let the APM Lambda Extension retrieve the actual keys from the AWS Secrets Manager. Follow the instructions below to set up the AWS Secrets Manager with the APM Lambda Extension.
+
+[[aws-lambda-secrets-manager-create-secret]]
+=== Step 1: Create a secret in the AWS Secrets Manager.
+https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html[Create a secret in the AWS Secrets Manager] for the {apm-guide-ref}/secret-token.html[APM Secret Token] or the {apm-guide-ref}/api-key.html[APM API key], depending on which one you prefer to use. Make sure to create the secret as a *Plaintext* typed secret and ensure it is created *in the same AWS region* as your target Lambda function that will use the secret. 
+
+We recommend using the AWS-managed encryption key `aws/secretsmanager`. However, you can optionally create and select a custom KMS key for encryption. Note that with a custom encryption key, you will need additional key permissions on your Lambda function (see <<aws-lambda-secrets-manager-permissions, Step 2>>).
+
+include::./secrets-manager/create-secret-widget.asciidoc[]
+
+Remember your chosen secret name. You will use the secret name as the value for the config options <<aws-lambda-config-secrets-manager-options>> when configuring your APM Lambda Extension.
+
+[[aws-lambda-secrets-manager-permissions]]
+=== Step 2: Add permissions to your AWS Lambda function
+For your Lambda function to be able to retrieve the authentication key from the AWS Secrets Manager, you need to provide the following permissions to your Lambda function.
+
+include::./secrets-manager/add-permissions-widget.asciidoc[]
+
+=== Step 3: Configure the APM Lambda extension
+Finally, you will need to configure the APM Lambda extension to use the secret from the Secrets Manager _instead of_ the value provided through <<aws-lambda-config-authentication-keys>>.
+
+Provide the name of the secret you created in <<aws-lambda-secrets-manager-create-secret, Step 1>> as the value for the <<aws-lambda-config-secrets-manager-options>> config option, respectively, depending on whether you want to use the {apm-guide-ref}/secret-token.html[APM Secret Token] or the {apm-guide-ref}/api-key.html[APM API key]. 
+
+The language-specific instructions describe how to set environment variables for configuring AWS Lambda for Elastic APM:
+
+* {apm-node-ref}/lambda.html#_step_3_configure_apm_on_aws_lambda[Configure APM on AWS Lambda - Node.js]
+* {apm-py-ref}/lambda-support.html#_step_3_configure_apm_on_aws_lambda[Configure APM on AWS Lambda - Python]
+* {apm-java-ref}/aws-lambda.html#_step_3_configure_apm_on_aws_lambda[Configure APM on AWS Lambda - Java]
+
+That's it. With the first invocation (cold start) of your Lambda function you should see a log message from the APM Lambda extension indicating that a secret from the secrets manager is used:
+
+[source, yml]
+----
+"Using the APM secret token retrieved from Secrets Manager."
+----

docs/secrets-manager/add-permissions-content.asciidoc

@@ -0,0 +1,191 @@
+// tag::console[]
+- In the Web Console of your AWS Lambda function navigate to `Configuration` -> `Permissions` and click on the link in the `Execution Role` section. 
+This will lead you to the Web Console for the corresponding IAM role.
+- Select `Create inline policy` in the `Add permission` dropdown button and select the `JSON` tab to define the following JSON-based policy:
++
+--
+[source,json]
+----
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Effect": "Allow",
+            "Action": [
+                "secretsmanager:GetSecretValue"
+            ],
+            "Resource": [
+                "THE_ARN_OF_YOUR_SECRET"
+            ]
+        },
+        { <1>
+            "Effect": "Allow",
+            "Action": [
+                "kms:Decrypt"
+            ],
+            "Resource": [
+                "THE_ARN_OF_YOUR_CUSTOM_KMS_KEY"
+            ]
+        }
+    ]
+}
+----
+<1> This permission is ONLY needed if you use a custom KMS encryption key for your secret
+--
+- Review and save the policy.
+
+// end::console[]
+
+// tag::cli[]
+- Create a policy file (`smElasticApmPolicy.json`) with the following content:
++
+--
+[source,json]
+----
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Effect": "Allow",
+            "Action": [
+                "secretsmanager:GetSecretValue"
+            ],
+            "Resource": [
+                "THE_ARN_OF_YOUR_SECRET"
+            ]
+        },
+        { <1>
+            "Effect": "Allow",
+            "Action": [
+                "kms:Decrypt"
+            ],
+            "Resource": [
+                "THE_ARN_OF_YOUR_CUSTOM_KMS_KEY"
+            ]
+        }
+    ]
+}
+----
+<1> This permission is ONLY needed if you use a custom KMS encryption key for your secret
+--
+- Retrieve the execution role name of your Lambda function with the following command:
++
+--
+[source,bash]
+----
+aws lambda get-function-configuration --function-name THE_NAME_OF_YOUR_FUNCTION | grep "Role"
+----
+--
+- Attach the new policy to the execution role:
++
+--
+[source,bash]
+----
+aws iam put-role-policy \
+  --role-name NAME_OR_ARN_OF_THE_EXECUTION_ROLE \
+  --policy-name SecretsManagerAPMKeyPolicy \
+  --policy-document file://smElasticApmPolicy.json
+----
+--
+// end::cli[]
+
+// tag::sam[]
+
+In your SAM `template.yml` file add the following policy to give the Lambda function access to the stored secret:
+
+[source,yml]
+----
+...
+AWSTemplateFormatVersion: '2010-09-09'
+Transform: 'AWS::Serverless-2016-10-31'
+Resources:
+  yourLambdaFunction:
+    Type: 'AWS::Serverless::Function'
+    Properties:
+      ...
+      Policies:
+      - Statement:
+        - Sid: SecretsManagerGetSecretValue
+          Effect: Allow
+          Action:
+            - secretsmanager:GetSecretValue
+          Resource: 'THE_ARN_OF_YOUR_SECRET'
+        - Sid: KMSDecrypt <1>
+          Effect: Allow
+          Action:
+            - kms:Decrypt
+          Resource: 'THE_ARN_OF_YOUR_CUSTOM_KMS_KEY'
+...
+----
+<1> This permission is ONLY needed if you use a custom KMS encryption key for your secret
+
+// end::sam[]
+
+// tag::serverless[]
+In your serverless file add the following policy to give the Lambda function access to the stored secret:
+[source,yml]
+----
+...
+provider:
+  iam:
+    role:
+      statements:
+        - Effect: Allow
+          Action: 'secretsmanager:GetSecretValue'
+          Resource: 'THE_ARN_OF_YOUR_SECRET'
+        - Effect: Allow <1>
+          Action: 'kms:Decrypt'
+          Resource: 'THE_ARN_OF_YOUR_CUSTOM_KMS_KEY'
+...
+----
+<1> This permission is ONLY needed if you use a custom KMS encryption key for your secret
+
+// end::serverless[]
+
+// tag::terraform[]
+Add the following policy and policy attachement resources to your terraform file to 
+give the Lambda function access to the stored secret:
+[source,terraform]
+----
+...
+resource "aws_iam_role" "lambda_role" {
+  // Here is your Lambda execution role
+  ...
+}
+
+resource "aws_iam_policy" "secrets_manager_elastic_apm_policy" {
+  name        = "secrets_manager_elastic_apm_policy"
+  description = "Allows the lambda function to access the APM authentication key stored in AWS Secrets Manager."
+
+  policy = <<EOF
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Action": [
+        "secretsmanager:GetSecretValue"
+      ],
+      "Effect": "Allow",
+      "Resource": "THE_ARN_OF_YOUR_SECRET"
+    },
+    { <1>
+      "Action": [
+        "kms:Decrypt"
+      ],
+      "Effect": "Allow",
+      "Resource": "THE_ARN_OF_YOUR_CUSTOM_KMS_KEY"
+    }
+  ]
+}
+EOF
+}
+
+resource "aws_iam_policy_attachment" "secrets_manager_elastic_apm_policy_attach" {
+  role      = aws_iam_role.lambda_role.name
+  policy_arn = aws_iam_policy.secrets_manager_elastic_apm_policy.arn
+}
+...
+----
+<1> This permission is ONLY needed if you use a custom KMS encryption key for your secret
+
+// end::terraform[]

docs/secrets-manager/add-permissions-widget.asciidoc

@@ -0,0 +1,99 @@
+++++
+<div class="tabs" data-tab-group="os">
+  <div role="tablist" aria-label="dependency">
+    <button role="tab"
+            aria-selected="true"
+            aria-controls="console-tab-add-permissions"
+            id="console-add-permissions">
+      AWS Web Console
+    </button>
+    <button role="tab"
+            aria-selected="false"
+            aria-controls="cli-tab-add-permissions"
+            id="cli-add-permissions"
+            tabindex="-1">
+      AWS CLI
+    </button>
+    <button role="tab"
+            aria-selected="false"
+            aria-controls="sam-tab-add-permissions"
+            id="sam-add-permissions"
+            tabindex="-1">
+      SAM
+    </button>
+    <button role="tab"
+            aria-selected="false"
+            aria-controls="serverless-tab-add-permissions"
+            id="serverless-add-permissions"
+            tabindex="-1">
+      Serverless
+    </button>
+    <button role="tab"
+            aria-selected="false"
+            aria-controls="terraform-tab-add-permissions"
+            id="terraform-add-permissions"
+            tabindex="-1">
+      Terraform
+    </button>
+  </div>
+  <div tabindex="0"
+      role="tabpanel"
+      id="console-tab-add-permissions"
+      name="lambda-tabpanel"
+      aria-labelledby="console-add-permissions">
+++++
+
+include::add-permissions-content.asciidoc[tag=console]
+
+++++
+  </div>
+  <div tabindex="0"
+      role="tabpanel"
+      id="cli-tab-add-permissions"
+      name="lambda-tabpanel"
+      aria-labelledby="cli-add-permissions"
+      hidden="">
+++++
+
+include::add-permissions-content.asciidoc[tag=cli]
+
+++++
+  </div>
+  <div tabindex="0"
+      role="tabpanel"
+      id="sam-tab-add-permissions"
+      name="lambda-tabpanel"
+      aria-labelledby="sam-add-permissions"
+      hidden="">
+++++
+
+include::add-permissions-content.asciidoc[tag=sam]
+
+++++
+  </div>
+  <div tabindex="0"
+      role="tabpanel"
+      id="serverless-tab-add-permissions"
+      name="lambda-tabpanel"
+      aria-labelledby="serverless-add-permissions"
+      hidden="">
+++++
+
+include::add-permissions-content.asciidoc[tag=serverless]
+
+++++
+  </div>
+  <div tabindex="0"
+      role="tabpanel"
+      id="terraform-tab-add-permissions"
+      name="lambda-tabpanel"
+      aria-labelledby="terraform-add-permissions"
+      hidden="">
+++++
+
+include::add-permissions-content.asciidoc[tag=terraform]
+
+++++
+  </div>
+</div>
+++++