[AppCheck] Firestore stops working after leaving the website idle overnight - AppCheck HTTP error 403 (appCheck/fetch-status-error) + infinite loop / high CPU usage

[anisabboud]

AppCheck problem

AppCheck works fine and enforced.
However, after leaving the webapp idle for a while (e.g., if I leave the tab open and go to sleep and put the laptop to sleep, then come to work the next day), Firestore stops working (as if it gets disconnected and doesn't reconnect or refresh AppCheck token).

Console error observed in production: @firebase/app-check: FirebaseError: AppCheck: Fetch server returned an HTTP error status. HTTP status: 403. (appCheck/fetch-status-error).

Console error observed on localhost: zone.js:1061 Unhandled Promise rejection: cancelled ; Zone: <root> ; Task: Promise.then ; Value: cancelled undefined

Environment

• Angular version: 14.0.2 (latest)
• Firebase SDK version: 9.8.3 (latest)
• AngularFire version: 7.4.1 (latest)
• Firebase Product: AppCheck

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[dconeybe]

Hello @anisabboud. I am working with the AppCheck team to investigate. I'll reply back once I have any updates.

The 403 error that you're seeing means that Firestore SDK has correctly identified that the token is expired and that it needs to get a new token. It has attempted to get a new AppCheck token, and AppCheck was not able to provide one (hence the error is coming from @firebase/app-check). So any Firestore operation after that will fail if AppCheck enforcement is ON. Interestingly, there was an issue with AppCheck and Flutter that saw the same error (Googlers can see b/230454903). The root cause might be the same.

[dconeybe]

@anisabboud Would it be possible for you to provide a minimal app to reproduce this issue, that we could use for debugging?

[anisabboud]

Hi @dconeybe, I don't currently have time to spin up a test app, but you could reproduce this issue on your own apps using Firestore listeners and AppCheck enforced. Just leave the tab open, put the computer to sleep, and check the console the next day. Try that a few days in a row.

The problem is that it's cumbersome to reproduce - I need to wait for the next day every time...
Today for instance, I encountered a different but related error when doing this - appCheck/fetch-network-error:
@firebase/app-check: FirebaseError: AppCheck: Fetch failed to connect to a network. Check Internet connection. Original error: Failed to fetch. (appCheck/fetch-network-error).

The app continued to function, but the console was outputting the message above in an infinite loop (~10,000 times within 5 minutes).

Attaching a partial log showing the infinite loop:

[dconeybe]

I've re-assigned this issue to my colleague @hsubox76 who has far more expertise in the realm of AppCheck. If this turns out to be a Firestore issue, feel free to assign it back to me.

[hsubox76]

I'm still not sure about the first error, but the second error doesn't seem to be related to the SDK itself, it seems like all the errors in the stack trace (in the last post) happen because the machine is offline, or at least the browser believes it is offline. If these happened after the computer woke up, then it seems like it either didn't reconnect to the internet properly, or the browser didn't recognize it.

As for the first error (the 403), it will be difficult to find the source of the problem if we can't reproduce it. All I can guess from my knowledge is that 403 errors like that generally happen because an invalid recaptcha token was sent to the app check exchange endpoint. A possible cause of an invalid recaptcha token might be that something about the sleep/wake process screws up the recaptcha process and causes it to yield an invalid recaptcha token, which is then sent to the app check endpoint, which returns the 403.

[anisabboud]

Hi @hsubox76 - thank you for the insights.
Note that the infinite loop of errors mentioned in my last message continues indefinitely (for more than 10 minutes, producing tens of thousands of errors until the console crashes), whereas the internet connection comes back a couple of seconds after waking the computer up.
I.e., the loop should stop when the internet comes back, but it doesn't, which seems like a bug.

[clive-h-townsend]

Just chiming in here that we are experiencing the same error across a broad swath of our users. A refresh solves the issue presumably related to the reCAPTCHA expiration. For reference, we are on reCAPTCHA Enterprise. The issue appears to arise after an overnight stale session.