Cannot catch requests.exceptions.ConnectionError with try except

Question:

It feels like I am slowly losing my sanity. I am unable to catch a connection error in a REST-API request. I read at least 20 similar questions on stackoverflow, tried every possible except statement I could think of and simplified the code as much as I could to rule out certain other libraries.

I am using Python 3.7 and requests 2.25.1. It is a very basic call to an API on my own server, which sometimes fails, but it only fails once in a while:

try:
    response = requests.get(url, headers=api_headers, auth=HTTPBasicAuth(username, password))
except requests.exceptions.ConnectionError:
     print("Connection error!")
     

I am sorry I cannot supply a full working example, as I am not connecting to an publicly accessible API, so I had to remove url, username and password.

Even though I try to catch the connection error, the script fails with following traceback:

Traceback (most recent call last):
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3connection.py", line 421, in connect
    tls_in_tls=tls_in_tls,
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3utilssl_.py", line 429, in ssl_wrap_socket
    sock, context, tls_in_tls, server_hostname=server_hostname
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3utilssl_.py", line 472, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libssl.py", line 412, in wrap_socket
    session=session
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libssl.py", line 850, in _create
    self.do_handshake()
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libssl.py", line 1108, in do_handshake
    self._sslobj.do_handshake()
TimeoutError: [WinError 10060] Ein Verbindungsversuch ist fehlgeschlagen, da die Gegenstelle nach einer bestimmten Zeitspanne nicht richtig reagiert hat, oder die hergestellte Verbindung war fehlerhaft, da der verbundene Host nicht reagiert hat

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesrequestsadapters.py", line 449, in send
    timeout=timeout
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3utilretry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3packagessix.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3connection.py", line 421, in connect
    tls_in_tls=tls_in_tls,
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3utilssl_.py", line 429, in ssl_wrap_socket
    sock, context, tls_in_tls, server_hostname=server_hostname
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3utilssl_.py", line 472, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libssl.py", line 412, in wrap_socket
    session=session
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libssl.py", line 850, in _create
    self.do_handshake()
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libssl.py", line 1108, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', TimeoutError(10060, 'Ein Verbindungsversuch ist fehlgeschlagen, da die Gegenstelle nach einer bestimmten Zeitspanne nicht richtig reagiert hat, oder die hergestellte Verbindung war fehlerhaft, da der verbundene Host nicht reagiert hat', None, 10060, None))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesrequestsapi.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesrequestssessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesrequestssessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesrequestsadapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', TimeoutError(10060, 'Ein Verbindungsversuch ist fehlgeschlagen, da die Gegenstelle nach einer bestimmten Zeitspanne nicht richtig reagiert hat, oder die hergestellte Verbindung war fehlerhaft, da der verbundene Host nicht reagiert hat', None, 10060, None))

I don’t understand how it is possible for the script to fail with requests.exceptions.ConnectionError if I am catching that very error?

If I understand that traceback correctly, the error is not thrown in my code, and therefore I am not able to catch it? All I see is python libraries like ssl.py and urllib and request, but not a line from my code. So how do I catch that?

Any help is highly appreciated!

EDIT (because this is not possible in a comment). @Thomas made a helpful comment to connect to httpstat.us:81 to debug. So I tried replacing my order_response = requests.get() call with response = requests.get("http://httpstat.us:81"). This is the exact block in my code:

try:
    order_response = requests.get(order_access_url, headers=api_headers, auth=HTTPBasicAuth(username, password))
    if order_response.status_code == 200:
        order_content = json.loads(order_response.text)
    else:
        order_content = ""
except requests.exceptions.ConnectionError:
    print("Connection error!")

If I am trying to connect to http://httpstat.us:81 it actually catches the error. If I intentionally not catch it, the error looks like it:

Traceback (most recent call last):
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3connection.py", line 170, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3utilconnection.py", line 96, in create_connection
    raise err
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3utilconnection.py", line 86, in create_connection
    sock.connect(sa)
TimeoutError: [WinError 10060] Ein Verbindungsversuch ist fehlgeschlagen, da die Gegenstelle nach einer bestimmten Zeitspanne nicht richtig reagiert hat, oder die hergestellte Verbindung war fehlerhaft, da der verbundene Host nicht reagiert hat

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libhttpclient.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libhttpclient.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libhttpclient.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libhttpclient.py", line 1016, in _send_output
    self.send(msg)
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libhttpclient.py", line 956, in send
    self.connect()
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3connection.py", line 200, in connect
    conn = self._new_conn()
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3connection.py", line 182, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x00000223F9B42860>: Failed to establish a new connection: [WinError 10060] Ein Verbindungsversuch ist fehlgeschlagen, da die Gegenstelle nach einer bestimmten Zeitspanne nicht richtig reagiert hat, oder die hergestellte Verbindung war fehlerhaft, da der verbundene Host nicht reagiert hat

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesrequestsadapters.py", line 449, in send
    timeout=timeout
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesurllib3utilretry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='httpstat.us', port=81): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000223F9B42860>: Failed to establish a new connection: [WinError 10060] Ein Verbindungsversuch ist fehlgeschlagen, da die Gegenstelle nach einer bestimmten Zeitspanne nicht richtig reagiert hat, oder die hergestellte Verbindung war fehlerhaft, da der verbundene Host nicht reagiert hat'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:Datencloud.bss-archery.comBSS_Twainmodulesorder_extracts_api.py", line 50, in create_order_analysis
    response = requests.get("http://httpstat.us:81")
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesrequestsapi.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesrequestsapi.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesrequestssessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesrequestssessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "C:UsersTilmanAppDataLocalProgramsPythonPython37libsite-packagesrequestsadapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='httpstat.us', port=81): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000223F9B42860>: Failed to establish a new connection: [WinError 10060] Ein Verbindungsversuch ist fehlgeschlagen, da die Gegenstelle nach einer bestimmten Zeitspanne nicht richtig reagiert hat, oder die hergestellte Verbindung war fehlerhaft, da der verbundene Host nicht reagiert hat'))

So I am still very confused because the last entry in the traceback is in fact the same, requests.exceptions.ConnectionError but it is not caught in my real world application. It is, however, raised by a different line in libsite-packagesrequestsadapters.py

Asked By: Tilman

||

Answers:

Okay, I could figure it out myself. Kind of.

A huge problem was that the traceback doesn’t point to the line of my code where the exception is raised. I still don’t know why that is and if this should be considered a bug in requests or not. But in any case: requests raises a ConnectionError in adapters.py but the origin is a protocol or socket error. This is line 497 in adapters.py:

except (ProtocolError, socket.error) as err:
    raise ConnectionError(err, request=request)

The TimeoutError: [WinError 10060] in the traceback actually points to a socket error.

From https://hstechdocs.helpsystems.com/manuals/globalscape/archive/cuteftp8/Socket_errors_10060_10061_10064_10065.htm:

A socket error in the 10060 range is a Winsock error. It is generally caused by either outgoing connection problems or connection problems on the host end.

That is why I wasn’t able to reproduce the error with httpstat.us.

The solution was to catch it as an OSError:

try:
    response = requests.get(url, headers=api_headers, auth=HTTPBasicAuth(username, password))
except OSError as e:
     print(e)

It’s a bit frustrating to be honest, as I still don’t know why ProtocolError or socket.error in requests that raises a ConnectionError needs to be caught with "OSError" but at this point, I am just glad I could find ANY solution.

Answered By: Tilman