AWS Lambda failing w/ RuntimeError: can't start new thread on 1.40.3

[thedanfields]

How do you use Sentry?

Self-hosted/on-premise

Version

1.40.3

Steps to Reproduce

Recently upgraded from 1.39.2 to 1.40.3.

Our AWS Lambda executors started failing with:

Traceback (most recent call last):
  < ... omitted ... >
  File "/var/task/sentry_sdk/integrations/threading.py", line 56, in sentry_start
    return old_start(self, *a, **kw)
  File "/var/lang/lib/python3.9/threading.py", line 899, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

Rolling back to 1.39.2 prevented the RuntimeError: can't start new thread error.

Expected Result

Lambda execution executed as expected.

Actual Result

A run time exception stemming from the sentry_sdk/threading/threading.py module:

File "/var/task/sentry_sdk/integrations/threading.py", line 56, in sentry_start
    return old_start(self, *a, **kw)
File "/var/lang/lib/python3.9/threading.py", line 899, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

[sentrivana]

So as said in the original thread I'm thinking this has to do with the SDK spawning a new thread by default in 1.40+ and possibly hitting some thread limits on AWS. We've already seen something similar in this issue, also AWS related.

If my suspicion is correct then this should make it go away:

sentry_sdk.init(
    ... # your usual stuff
    _experiments={
        "enable_metrics": False,
    }
) 

@thedanfields Could you give this a shot and see if it makes a difference?

[gksb88]

Hi @sentrivana

We upgraded to 1.43.0 and are still seeing this issue with the _experiments={"enable_metrics": False} fix.

The issue happens specifically when a lot of lambdas are writing to s3 or reading from it.

We are on 1.43.0.

[kerenkhatiwada]

Hi @sentrivana, can you please also confirm which previous version of the SDK this was working in?

[sentrivana]

Hey folks, thanks for following up.

@gksb88 Which SDK version did you upgrade from? Are you creating your own threads in your app?

@kerenkhatiwada According to the OP,

Rolling back to 1.39.2 prevented the RuntimeError: can't start new thread error.

In this case, the most likely culprit from the changes between the two versions (1.39.2 and 1.40.3) was turning metrics on by default, and by extension, starting the background metrics flusher thread. In @gksb88's case though this might be a different issue since turning metrics off doesn't make a difference.