V2 Functions crashing when deployed with 12.6.1

[milo-]

After deploying using firebase-tools 12.6.1, our functions are crashing at runtime with the following stacktrace:

Error: TypeError: Cannot read properties of undefined (reading 'headers')
    at /layers/google.nodejs.yarn/yarn_modules/node_modules/firebase-functions/lib/v2/trace.js:12:68
    at /layers/google.nodejs.functions-framework/functions-framework/node_modules/@google-cloud/functions-framework/build/src/function_wrappers.js:141:25
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at console.error (/layers/google.nodejs.yarn/yarn_modules/node_modules/firebase-functions/lib/logger/compat.js:31:23)
    at sendCrashResponse (/layers/google.nodejs.functions-framework/functions-framework/node_modules/@google-cloud/functions-framework/build/src/logger.js:27:17)
    at sendResponse (/layers/google.nodejs.functions-framework/functions-framework/node_modules/@google-cloud/functions-framework/build/src/invoker.js:37:40)
    at /layers/google.nodejs.functions-framework/functions-framework/node_modules/@google-cloud/functions-framework/build/src/function_wrappers.js:34:40
    at bound (node:domain:433:15)
    at runBound (node:domain:444:12)
    at /layers/google.nodejs.functions-framework/functions-framework/node_modules/@google-cloud/functions-framework/build/src/function_wrappers.js:142:60
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

We're also crashing in our own code when trying to read request.rawBody.toString() as rawBody is undefined. It looks like something is wrong with the request objects being passed to the http functions.

We saw a very similar issue in 12.5.0 (#6290)

It looks like with this PR #6376 the issue might have been reintroduced.

@blidd-google can you take a look at this please?

[REQUIRED] Environment info

firebase-functions v2
Node 18.12.1

firebase-tools:

12.6.1

Platform:

[REQUIRED] Test case

Deploy a V2 http function with 12.6.1

Redeploy with 12.4.8 and functions work correctly

[REQUIRED] Steps to reproduce

Deploy a V2 http function with 12.6.1

[REQUIRED] Expected behaviour

It should not crash, correct properties should be defined on the request/response object

[REQUIRED] Actual behaviour

Crash, request object properties are undefined

[google-oss-bot]

This issue does not seem to follow the issue template. Make sure you provide all the required information.

[aalej]

Hey @milo-, sorry to hear you’re encountering this issue. I’m currently trying to reproduce the error you mentioned but so far I am unable to replicate this. I tried deploying a mixture of v1 and v2 HTTP functions:

const functionv1 = require("firebase-functions/v1");
const functionv2 = require("firebase-functions/v2");
const logger = require("firebase-functions/logger");

exports.helloWorld1 = functionv1.https.onRequest((request, response) => {
  const rawBody = request.rawBody.toString();
  logger.info(`rawBody: ${rawBody}`, { structuredData: true });
  response.send(`Hello from Firebase v1! ${rawBody}`);
});

exports.helloWorld2 = functionv2.https.onRequest((request, response) => {
  const rawBody = request.rawBody.toString();
  logger.info(`rawBody: ${rawBody}`, { structuredData: true });
  response.send(`Hello from Firebase v2 2! ${rawBody}`);
});

exports.helloWorld3 = functionv2.https.onRequest((request, response) => {
  const rawBody = request.rawBody.toString();
  logger.info(`rawBody: ${rawBody}`, { structuredData: true });
  response.send(`Hello from Firebase v2 3! ${rawBody}`);
});

Making a POST request to any of the HTTP functions above does not raise any errors. In case I’m missing anything here, please let me know.

Could you provide additional information here such as if you’re deploying a mixture of v1 and v2 functions or just multiple v2 functions? By any chance, if you can provide a sample code snippet of the function(s) you’re deploying, we would highly appreciate it.

[milo-]

Hi @aalej, yep we're deploying ~60 functions, 50/50 mixture of v1 functions and v2 functions.

We were seeing this issue in the following v2 functions:

• beforeUserSignedIn
• onSchedule
• onRequest

All these functions are group functions. Ill try and get a smaller repro for you.

[lelimit]

I also got problems at runtime after deploying onCall v2 functions with with 12.6.1.
TypeError: res.on is not a function

[milo-]

I've been unable to create a small reproduction. But can confirm this issue happens every time we deploy with 12.6.1.

Deploying the individual function fixes the issue, but when all functions are deployed together, the runtime error res.headers is undefined starts getting thrown again.

@blidd-google I've been digging into the source and the logs. Something that seems strange is there's a v1 and a v2 scraper, so I'd expect a source token for v1, source token for v2. However during deployment, it logs 4 different source tokens:

3 x UUID source tokens
1 x 384 character source token

[blidd-google]

Hi @milo-, very sorry for all the disruptions you've run into. I'm investigating the issue now and am also having trouble creating a repro of the bug you've described, but I'll keep this thread updated.

[blidd-google]

A couple of follow up questions:

• Is the correct grouped function being invoked before crashing at runtime due to the missing request object?
• In addition to grouped functions, are you using multiple codebases to organize your functions? This would explain why more than 2 source tokens.
• What is the shape of the HTTP request you are sending to the onRequest function?
• Can you provide the logs from running the deploy command with the --debug flag?