From time to time "Connection reset by peer" error while sending event to the server

[sliwinski-milosz]

Environment

SaaS (https://sentry.io/)

Version

No response

Steps to Reproduce

I tried to debug this issue however I am not 100% sure what is the root cause. I think the problem is related to:
urllib3/urllib3#944 (comment)
I have applied following workaround and it fixed the problem:
psf/requests#4937 (comment)

It seems that for the Python client, the keep alive option is disabled by default. Maybe this is causing connection problems.
Even that this looks like a Python client issue, and that the workaround on the client side worked fine - I wonder if there is a way to fix this on the server side. This is why I am opening the issue here.

Maybe changing the idle connection timeout on the server side would help. I would appreciate your expertise on that part.

It is not easy to reproduce this error. I tried to reproduce it from local machine by sending events every x seconds (1, 10, 60, 300, 1800). I have reproduced it only twice using 1800 secs delay between two events. However often, even for 1800 it was working fine.
After enabling DEBUG logs on client side, I have realised that this issue happens few times a day for my service. I am happy to gather some more logs if needed.

Below I am sending full Python traceback logs for the issue:

[2021-09-01 15:34:10,901] [] [] [DEBUG] [] Sending event, type:null level:error event_id:xxx project:xxx host:xxx.ingest.sentry.io
[sentry] ERROR: Internal error in sentry_sdk
Traceback (most recent call last):
File "/usr/app/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
File "/usr/app/venv/lib/python3.8/site-packages/newrelic/hooks/external_urllib3.py", line 32, in _nr_wrapper_make_request_
    return wrapped(*args, **kwargs)
File "/usr/app/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/app/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
File "/usr/app/venv/lib/python3.8/site-packages/sentry_sdk/integrations/stdlib.py", line 102, in getresponse
    rv = real_getresponse(self, *args, **kwargs)
File "/usr/app/venv/lib/python3.8/site-packages/newrelic/hooks/external_httplib.py", line 77, in httplib_getresponse_wrapper
    return wrapped(*args, **kwargs)
File "/usr/lib/python3.8/http/client.py", line 1322, in getresponse
    response.begin()
File "/usr/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
File "/usr/lib/python3.8/http/client.py", line 264, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
File "/usr/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
File "/usr/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/app/venv/lib/python3.8/site-packages/sentry_sdk/transport.py", line 339, in send_event_wrapper
    self._send_event(event)
File "/usr/app/venv/lib/python3.8/site-packages/sentry_sdk/transport.py", line 245, in _send_event
    self._send_request(
File "/usr/app/venv/lib/python3.8/site-packages/sentry_sdk/transport.py", line 179, in _send_request
    response = self._pool.request(
File "/usr/app/venv/lib/python3.8/site-packages/urllib3/request.py", line 78, in request
    return self.request_encode_body(
File "/usr/app/venv/lib/python3.8/site-packages/urllib3/request.py", line 170, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
File "/usr/app/venv/lib/python3.8/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
File "/usr/app/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
File "/usr/app/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
File "/usr/app/venv/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
File "/usr/app/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
File "/usr/app/venv/lib/python3.8/site-packages/newrelic/hooks/external_urllib3.py", line 32, in _nr_wrapper_make_request_
    return wrapped(*args, **kwargs)
File "/usr/app/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/app/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
File "/usr/app/venv/lib/python3.8/site-packages/sentry_sdk/integrations/stdlib.py", line 102, in getresponse
    rv = real_getresponse(self, *args, **kwargs)
File "/usr/app/venv/lib/python3.8/site-packages/newrelic/hooks/external_httplib.py", line 77, in httplib_getresponse_wrapper
    return wrapped(*args, **kwargs)
File "/usr/lib/python3.8/http/client.py", line 1322, in getresponse
    response.begin()
File "/usr/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
File "/usr/lib/python3.8/http/client.py", line 264, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
File "/usr/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
File "/usr/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2021-09-01 15:34:10,903] [] [] [ERROR] [] Internal error in sentry_sdk Traceback (most recent call last):   File "/usr/app/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen     httplib_response = self._make_request(   File "/usr/app/venv/lib/python3.8/site-packages/newrelic/hooks/external_urllib3.py", line 32, in _nr_wrapper_make_request_     return wrapped(*args, **kwargs)   File "/usr/app/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 445, in _make_request     six.raise_from(e, None)   File "<string>", line 3, in raise_from   File "/usr/app/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 440, in _make_request     httplib_response = conn.getresponse()   File "/usr/app/venv/lib/python3.8/site-packages/sentry_sdk/integrations/stdlib.py", line 102, in getresponse     rv = real_getresponse(self, *args, **kwargs)   File "/usr/app/venv/lib/python3.8/site-packages/newrelic/hooks/external_httplib.py", line 77, in httplib_getresponse_wrapper     return wrapped(*args, **kwargs)   File "/usr/lib/python3.8/http/client.py", line 1322, in getresponse     response.begin()   File "/usr/lib/python3.8/http/client.py", line 303, in begin     version, status, reason = self._read_status()   File "/usr/lib/python3.8/http/client.py", line 264, in _read_status     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")   File "/usr/lib/python3.8/socket.py", line 669, in readinto     return self._sock.recv_into(b)   File "/usr/lib/python3.8/ssl.py", line 1241, in recv_into     return self.read(nbytes, buffer)   File "/usr/lib/python3.8/ssl.py", line 1099, in read     return self._sslobj.read(len, buffer) ConnectionResetError: [Errno 104] Connection reset by peer  During handling of the above exception, another exception occurred:  Traceback (most recent call last):   File "/usr/app/venv/lib/python3.8/site-packages/sentry_sdk/transport.py", line 339, in send_event_wrapper     self._send_event(event)   File "/usr/app/venv/lib/python3.8/site-packages/sentry_sdk/transport.py", line 245, in _send_event     self._send_request(   File "/usr/app/venv/lib/python3.8/site-packages/sentry_sdk/transport.py", line 179, in _send_request     response = self._pool.request(   File "/usr/app/venv/lib/python3.8/site-packages/urllib3/request.py", line 78, in request     return self.request_encode_body(   File "/usr/app/venv/lib/python3.8/site-packages/urllib3/request.py", line 170, in request_encode_body     return self.urlopen(method, url, **extra_kw)   File "/usr/app/venv/lib/python3.8/site-packages/urllib3/poolmanager.py", line 375, in urlopen     response = conn.urlopen(method, u.request_uri, **kw)   File "/usr/app/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen     retries = retries.increment(   File "/usr/app/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 532, in increment     raise six.reraise(type(error), error, _stacktrace)   File "/usr/app/venv/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise     raise value.with_traceback(tb)   File "/usr/app/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen     httplib_response = self._make_request(   File "/usr/app/venv/lib/python3.8/site-packages/newrelic/hooks/external_urllib3.py", line 32, in _nr_wrapper_make_request_     return wrapped(*args, **kwargs)   File "/usr/app/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 445, in _make_request     six.raise_from(e, None)   File "<string>", line 3, in raise_from   File "/usr/app/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 440, in _make_request     httplib_response = conn.getresponse()   File "/usr/app/venv/lib/python3.8/site-packages/sentry_sdk/integrations/stdlib.py", line 102, in getresponse     rv = real_getresponse(self, *args, **kwargs)   File "/usr/app/venv/lib/python3.8/site-packages/newrelic/hooks/external_httplib.py", line 77, in httplib_getresponse_wrapper     return wrapped(*args, **kwargs)   File "/usr/lib/python3.8/http/client.py", line 1322, in getresponse     response.begin()   File "/usr/lib/python3.8/http/client.py", line 303, in begin     version, status, reason = self._read_status()   File "/usr/lib/python3.8/http/client.py", line 264, in _read_status     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")   File "/usr/lib/python3.8/socket.py", line 669, in readinto     return self._sock.recv_into(b)   File "/usr/lib/python3.8/ssl.py", line 1241, in recv_into     return self.read(nbytes, buffer)   File "/usr/lib/python3.8/ssl.py", line 1099, in read     return self._sslobj.read(len, buffer) urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

Dependencies versions on client side:

ubuntu:18.04 - Python 3.8.0
sentry-sdk==1.3.1
urllib3==1.26.6

Expected Result

Client is able to successfully send all events to the server.

Actual Result

Due to connectivity problems, some events fail on the client side.

[getsentry-release]

Routing to @getsentry/owners-ingest for triage. ⏲️

[untitaker]

I believe this is a longer-standing issue with the Python SDK, but the web platform folks may know if we have similar issues in other SDKs (in that case ingest-team should investigate it)

[github-actions]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[sliwinski-milosz]

I think this issue is a major one and has to be fixed in the near future.

[sliwinski-milosz]

Are there any updates related to this specific issue? The idea shared here was to expose socket_options.

@sl0thentr0py could you please tell me if that is still preferable way to approach this issue or maybe you have some other ideas? :)