Retrieve APM Server info with a reverse proxy

[jlvoiseux]

Motivation / Summary

This pull request aims to implement the changes suggested as part of the high-level code review performed in #124. It replaces the manual forwarding of the APM agent's request for APM server info by a reverse proxy-based implementation. Go's ReverseProxy implements the desired behavior out of the box.

Resolves #138

Validation

This PR passes all pre-existing unit-tests.
Testing on a real Lambda function with the Python agent also shows that APM server info is successfully fetched :

x-amd64	State: Ready	Events: [INVOKE,SHUTDOWN]
{"@timestamp":"2022-03-22T12:16:46.559Z","ecs.version":"1.6.0","event.dataset":"apm-lambda-extension","log.level":"debug","message":"Received event."}
{"@timestamp":"2022-03-22T12:16:46.559Z","ecs.version":"1.6.0","event.dataset":"apm-lambda-extension","log.level":"debug","message":"{\n\t\"eventType\": \"INVOKE\",\n\t\"deadlineMs\": 1647951426558,\n\t\"requestId\": \"d85b1cc8-69d0-4ae1-b542-18bffa268402\",\n\t\"invokedFunctionArn\": \"arn:aws:lambda:eu-central-1:627286350134:function:lambda-sample-dev-jlvoiseux\",\n\t\"tracing\": {\n\t\t\"type\": \"X-Amzn-Trace-Id\",\n\t\t\"value\": \"Root=1-6239be2d-58c8e0f51767e95a017bbcba;Parent=3e6caf5c46d4d153;Sampled=0\"\n\t}\n}"}
{"@timestamp":"2022-03-22T12:16:46.559Z","ecs.version":"1.6.0","event.dataset":"apm-lambda-extension","log.level":"debug","message":"Received log event platform.logsSubscription"}
[DEBUG]	2022-03-22T12:16:46.559Z	d85b1cc8-69d0-4ae1-b542-18bffa268402	flushing elasticapm
[INFO]	2022-03-22T12:16:46.580Z	d85b1cc8-69d0-4ae1-b542-18bffa268402	Fetched APM Server version 8.1.0
[DEBUG]	2022-03-22T12:16:46.581Z	d85b1cc8-69d0-4ae1-b542-18bffa268402	forced flush
{"@timestamp":"2022-03-22T12:16:46.599Z","ecs.version":"1.6.0","event.dataset":"apm-lambda-extension","log.level":"debug","message":"Received log event platform.start"}
{"@timestamp":"2022-03-22T12:16:46.600Z","ecs.version":"1.6.0","event.dataset":"apm-lambda-extension","log.level":"debug","message":"Received log event platform.extension"}
{"@timestamp":"2022-03-22T12:16:46.619Z","ecs.version":"1.6.0","event.dataset":"apm-lambda-extension","log.level":"debug","message":"Adding agent data to buffer to be sent to apm server"}
[DEBUG]	2022-03-22T12:16:46.620Z	d85b1cc8-69d0-4ae1-b542-18bffa268402	Sent request, url=http://localhost:8200/intake/v2/events size=0.55kb status=202
{"@timestamp":"2022-03-22T12:16:46.639Z","ecs.version":"1.6.0","event.dataset":"apm-lambda-extension","log.level":"debug","message":"Sending data chunk to APM Server"}
[DEBUG]	2022-03-22T12:16:46.640Z	d85b1cc8-69d0-4ae1-b542-18bffa268402	done flushing elasticapm
{"@timestamp":"2022-03-22T12:16:46.668Z","ecs.version":"1.6.0","event.dataset":"apm-lambda-extension","log.level":"debug","message":"Received log event platform.runtimeDone"}
{"@timestamp":"2022-03-22T12:16:46.668Z","ecs.version":"1.6.0","event.dataset":"apm-lambda-extension","log.level":"info","message":"Received runtimeDone event for this function invocation"}
{"@timestamp":"2022-03-22T12:16:46.668Z","ecs.version":"1.6.0","event.dataset":"apm-lambda-extension","log.level":"debug","message":"Received runtimeDone signal"}
{"@timestamp":"2022-03-22T12:16:46.771Z","ecs.version":"1.6.0","event.dataset":"apm-lambda-extension","log.level":"debug","message":"Transport status set to healthy"}
{"@timestamp":"2022-03-22T12:16:46.771Z","ecs.version":"1.6.0","event.dataset":"apm-lambda-extension","log.level":"debug","message":"APM server response body: "}
{"@timestamp":"2022-03-22T12:16:46.771Z","ecs.version":"1.6.0","event.dataset":"apm-lambda-extension","log.level":"debug","message":"APM server response status code: 202"}
{"@timestamp":"2022-03-22T12:16:46.771Z","ecs.version":"1.6.0","event.dataset":"apm-lambda-extension","log.level":"debug","message":"Received signal that function has completed, not processing any more agent data"}
{"@timestamp":"2022-03-22T12:16:46.771Z","ecs.version":"1.6.0","event.dataset":"apm-lambda-extension","log.level":"debug","message":"Flush started - Checking for agent data"}
{"@timestamp":"2022-03-22T12:16:46.771Z","ecs.version":"1.6.0","event.dataset":"apm-lambda-extension","log.level":"debug","message":"Flush ended - No agent data on buffer"}
{"@timestamp":"2022-03-22T12:16:46.771Z","ecs.version":"1.6.0","event.dataset":"apm-lambda-extension","log.level":"info","message":"Waiting for next event..."}
END RequestId: d85b1cc8-69d0-4ae1-b542-18bffa268402
REPORT RequestId: d85b1cc8-69d0-4ae1-b542-18bffa268402	Duration: 213.49 ms	Billed Duration: 214 ms	Memory Size: 128 MB	Max Memory Used: 77 MB	Init Duration: 496.78 ms

[ghost]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-03-23T11:00:57.641+0000

• Duration: 7 min 13 sec

Test stats 🧪

Test	Results
Failed	0
Passed	240
Skipped	6
Total	246

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[jlvoiseux]

Converted PR to draft after a failure to retrieve APM Server info with the Node agent. Investigating the cause of this failure.

{
    "log.level": "debug",
    "@timestamp": "2022-03-22T13:57:35.515Z",
    "log": {
        "logger": "elastic-apm-node"
    },
    "event.module": "apmclient",
    "ecs": {
        "version": "1.6.0"
    },
    "message": "verfetch: unexpected status from APM Server information endpoint: 404"
}

[jlvoiseux]

The APM Server info request now works on all 3 supported languages :

Java (11 - Corretto)

2022-03-22 18:10:15,887 [elastic-apm-server-healthcheck] INFO  co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {
    "build_date": ...",
    "build_sha": "...",
    "publish_ready": true,
    "version": "8.1.0"
}

Python (3.9)

[INFO]	2022-03-22T18:15:08.885Z	f66b3824-a3cb-43a0-998a-91fcab2943eb	Fetched APM Server version 8.1.0

Node (14.X)

{
    "log.level": "debug",
    "@timestamp": "2022-03-22T18:14:22.802Z",
    "log": {
        "logger": "elastic-apm-node"
    },
    "event.module": "apmclient",
    "apmServerVersion": "8.1.0",
    "ecs": {
        "version": "1.6.0"
    },
    "message": "fetched APM Server version"
}

Moving the PR back to Ready for review

[estolfo]

Just one typo