Add X-Ray daemon

[dominikschubert]

Adds the xray daemon as a dependency and starts the daemon asynchronously.

The daemon is stopped and recorded segments are flushed when init shuts down (see signal handler / shutdown funcs in sandbox).

It's not yet ideal, but it provides parity with the old xray implementation. For proper active & passive tracing with additional generated (sub-)segments we'll need to add a few more things

• Generate & publish traces in init. There's actually already some things we can re-use here
• Properly use provided credentials for service calls
• Generate the root trace segment in the lambda service.

Other changes:

• Added a basic LOCALSTACK_INIT_LOG_LEVEL internal variable for us to be able to configure the log level in the init binary without re-compiling. I don't think this will be the final setup, since I still want to rework the general log setup here, but it's kinda useful in the meantime.

[joe4dev]

LGTM 👍

just minor comments but nothing blocking.

[joe4dev]

When do we adjust that to avoid log spamming?
The daemon itself should be stable, this might just be interesting for debugging.

[joe4dev]

It periodically logs Skipped telemetry data as no segments found or the number of traces sent.

[joe4dev]

The PR description is not up-to-date anymore after the change we introduced for updating the X-Ray TraceId with every request.

[joe4dev]

Could it be that the runtime starts before the X-Ray daemon due to async runDaemon(d) given the warning: Subsegment lambda discarded due to Lambda worker still initializing 🤔 (probably not blocking now but good to follow up later)

2023-03-20 23:02:07 2023-03-20T22:02:07Z [Debug] Batch size: 10
2023-03-20 23:02:07 2023-03-20T22:02:07Z [Info] Starting proxy http server on 127.0.0.1:2000
2023-03-20 23:02:07 time="2023-03-20T22:02:07Z" level=debug msg="Hot reloading disabled." func=main.RunHotReloadingListener file="/Users/joe/Projects/LocalStack/lambda-runtime-init/cmd/localstack/awsutil.go:150"
2023-03-20 23:02:07 time="2023-03-20T22:02:07Z" level=debug msg="Runtime API Server listening on 127.0.0.1:9001" func="go.amzn.com/lambda/rapi.(*Server).Listen" file="/Users/joe/Projects/LocalStack/lambda-runtime-init/lambda/rapi/server.go:93"
2023-03-20 23:02:07 time="2023-03-20T22:02:07Z" level=info msg="Configure environment for Init Caching." func="go.amzn.com/lambda/rapid.(*rapidContext).acceptStartRequestForInitCaching" file="/Users/joe/Projects/LocalStack/lambda-runtime-init/lambda/rapid/start.go:391"
2023-03-20 23:02:07 time="2023-03-20T22:02:07Z" level=info msg="extensionsDisabledByLayer(/opt/disable-extensions-jwigqn8j) -> stat /opt/disable-extensions-jwigqn8j: no such file or directory" func=go.amzn.com/lambda/rapid.extensionsDisabledByLayer file="/Users/joe/Projects/LocalStack/lambda-runtime-init/lambda/rapid/start.go:363"
2023-03-20 23:02:07 time="2023-03-20T22:02:07Z" level=debug msg="Received RUNNING" func="go.amzn.com/lambda/rapidcore.(*Server).Init" file="/Users/joe/Projects/LocalStack/lambda-runtime-init/lambda/rapidcore/server.go:533"
2023-03-20 23:02:07 time="2023-03-20T22:02:07Z" level=warning msg="Cannot list external agents" func=go.amzn.com/lambda/agents.ListExternalAgentPaths file="/Users/joe/Projects/LocalStack/lambda-runtime-init/lambda/agents/agent.go:71" error="open /opt/extensions: no such file or directory"
2023-03-20 23:02:07 time="2023-03-20T22:02:07Z" level=debug msg="Preregister runtime" func=go.amzn.com/lambda/rapid.doInit file="/Users/joe/Projects/LocalStack/lambda-runtime-init/lambda/rapid/start.go:189"
2023-03-20 23:02:07 time="2023-03-20T22:02:07Z" level=debug msg="Start runtime" func=go.amzn.com/lambda/rapid.doInit file="/Users/joe/Projects/LocalStack/lambda-runtime-init/lambda/rapid/start.go:222"
2023-03-20 23:02:10 [INFO]      2023-03-20T22:02:10.899Z                successfully patched module requests
2023-03-20 23:02:10 [INFO]      2023-03-20T22:02:10.925Z                successfully patched module botocore
2023-03-20 23:02:10 [INFO]      2023-03-20T22:02:10.948Z                successfully patched module sqlite3
2023-03-20 23:02:11 [INFO]      2023-03-20T22:02:11.032Z                Found credentials in environment variables.
2023-03-20 23:02:11 [WARNING]   2023-03-20T22:02:11.290Z                Subsegment lambda discarded due to Lambda worker still initializing
2023-03-20 23:02:12 [WARNING]   2023-03-20T22:02:12.044Z                No subsegment to end.

[dominikschubert]

Subsegment lambda discarded due to Lambda worker still initializing

Seems to be related to this issue here: aws/aws-xray-sdk-python#185