Description
Expected Behavior
When I make requests using the Fetch API and using Tracer, the X-Amzn-Trace-Id header should be constructed and forwarded along with the request so that I can get a full trace for a request.
For example, if I have an API Gateway A -> Lambda A -> API Gateway B -> Lambda B, and the first function in the chain is instrumented with Tracer, we should be able to see a trace like this:
This is already supported when using the global http and https modules and their derivatives (i.e. axios) but doesn't work in our implementation when using Tracer with fetch.
Current Behavior
Currently the setup described above generates two disconnected traces, one for each API Gateway -> Lambda pair.
Code snippet
import { Logger } from "@aws-lambda-powertools/logger";
import { Tracer } from "@aws-lambda-powertools/tracer";
const logger = new Logger();
const tracer = new Tracer();
export const handler = async (event: { userId: number }) => {
const parent = tracer.getSegment();
const segment = parent?.addNewSubsegment("### functionA handler");
segment && tracer.setSegment(segment);
await fetch(`${process.env.API_URL}/functionB`, {
method: "POST",
body: JSON.stringify({ userId: event.userId }),
});
segment?.close();
parent && tracer.setSegment(parent);
return {
statusCode: 200,
};
};Steps to Reproduce
- Deploy the infrastructure below
/// <reference path="./.sst/platform/config.d.ts" />
export default $config({
app(input) {
return {
name: "missing-trace",
removal: input?.stage === "production" ? "retain" : "remove",
home: "aws",
};
},
async run() {
$transform(sst.aws.Function, (args) => {
args.permissions = [
{
actions: [
"xray:PutTraceSegments",
"xray:PutTelemetryRecords",
"xray:GetSamplingRules",
"xray:GetSamplingTargets",
"xray:GetSamplingStatisticSummaries",
],
resources: ["*"],
},
];
args.transform = {
function(args, opts, name) {
args.tracingConfig = {
mode: "Active",
};
},
};
});
const apiB = new sst.aws.ApiGatewayV1("ApiB", {
transform: {
stage(args, opts, name) {
args.xrayTracingEnabled = true;
},
},
});
apiB.route("POST /functionB", {
handler: "src/functionB.handler",
runtime: "nodejs22.x",
timeout: "30 seconds",
memory: "256 MB",
logging: {
retention: "1 day",
},
});
apiB.deploy();
const apiA = new sst.aws.ApiGatewayV1("ApiA", {
transform: {
stage(args, opts, name) {
args.xrayTracingEnabled = true;
},
},
});
apiA.route("POST /functionA", {
handler: "src/functionA.handler",
runtime: "nodejs22.x",
timeout: "30 seconds",
memory: "256 MB",
logging: {
retention: "1 day",
},
environment: {
API_URL: apiB.url,
},
});
apiA.deploy();
},
});- Make a request to the first API Gateway endpoint (i.e.
http POST https://api-id.execute-api.eu-west-1.amazonaws.com/stage/functionA userId=1 - Observe the traces
Possible Solution
Our implementation of the fetch module diverges significantly from the one in aws-xray-sdk-node-fetch. Their implementation relies on monkey patching, which as far as I can tell would only work with CJS.
On our side we decided (#1619) to instead use the node:diagnostics_channel which is the recommended way to instrument requests made with fetch.
During the implementation, we followed the types present in @types/node which suggest it's not possible to modify the Request object when instrumenting.
In reality however request object in the message has a addHeader() method that can be used to forward the X-Amzn-Trace-Id.
To make this work, we should construct the header to include the Root, Parent, and Sampled fields and add it to the request. I made a PoC of this and I already got it working. The screenshot at the top of the issue comes from a request instrumented with Tracer using Fetch.
Powertools for AWS Lambda (TypeScript) version
latest
AWS Lambda function runtime
22.x
Packaging format used
npm