Is there a way to globally override requests' timeout setting?

Question

We have usages of the requests library littered throughout our project. Recently we came across a bug in one of our destinations where it froze mid transaction, and decided to just hold the connection open.

Naturally, our application followed suit.

Is there a environment variable, or some other way to set the timeout? Even if it’s significant (say, 30 seconds) it should be enough to stop the entire works from stopping because of one service. If possible, it should be global so that I don’t have to find every single use, and so that people can’t forget to add it in the future.

Asked By: Shadow

||

Source

Answer 1

You can certainly do it per call with the timeout parameter:

requests.get('http://www.google.com', timeout=10)

The timeout parameter specifies the number of seconds or a tuple for (connection timeout, read timout). You can read more about it here.

If you want to make this global, the easiest way is to refactor all of the calls into a wrapper class and make sure that everyone is using the wrapper class to call this API endpoint. I did a quick check of the requests code and did not see the use of a global override for timeout since the default behavior is to wait for data.

Answered By: 2ps

Answer 2

Unfortunately, looking at the code, there is no possibility to set a global default value. I was kinda surprised by that, as I would expect that to be quite common use case. If you start a feature request, please let me know (e.g. in comments to this post).

The reason for that is that methods like get(...), post(...), etc are all just thin wrappers over Session.request(...) method (requests.get(...) creates new one-shot session, just for a single request). That method takes timeout as argument, and does not inspect Session internals for a value if there is no timeout argument, so you always have to put it there manually, like 2ps proposed in his answer.

Sources:

Revised on master on 31.08.2020. Line numbers have changed, but methods stayed the same. The answer stays the same.

requests/__init__.py – import API to package scope, to provide requests.get(...)-like utilities
requests.api – API module that is imported in point above; usess one-shot sessions
requests.sessions – Session implementation
- line 337 starts Session class
- line 463 starts request(...) method
- line 526 actually uses timeout parameter
- line 534 start get(...) method

PS. See this pull request. Disclaimer: it’s mine.

Answered By: Filip Malczak

Answer 3

Instead you could inherit the requests.Session class and rewrite request function, like this.

HTTP_TIMEOUT = 30

class TimeoutRequestsSession(requests.Session):
    def request(self, *args, **kwargs):
        kwargs.setdefault('timeout', HTTP_TIMEOUT)
        return super(TimeoutRequestsSession, self).request(*args, **kwargs)

session = TimeoutRequestsSession()
session.get('https://www.google.com') # connection timeout is default 30s

Answered By: codeskyblue

Answer 4

Two ways to make this happen. It involves some dirty monkey patching. Tested on python 3.6.

Put this somewhere in a file like main.py, an init.py or urls.py and make sure it is called.

Option 1: patch the requests method

import requests

def request_patch(slf, *args, **kwargs):
    print("Fix called")
    timeout = kwargs.pop('timeout', 2)
    return slf.request_orig(*args, **kwargs, timeout=timeout)

setattr(requests.sessions.Session, 'request_orig', requests.sessions.Session.request)
requests.sessions.Session.request = request_patch

Option 2: Patch the Session class

import requests

class SessionTimeoutFix(requests.Session):

    def request(self, *args, **kwargs):
        print("Fix called")
        timeout = kwargs.pop('timeout', 2)
        return super().request(*args, **kwargs, timeout=timeout)

requests.sessions.Session = SessionTimeoutFix

Answered By: Roel Kramer

Answer 5

Why not?:

    for method in ("get", "options", "head", "post", "put", "patch", "delete"):
        setattr(
            session,
            method,
            functools.partial(getattr(session, method), timeout=timeout),
        )

Answered By: vicusbass

Answer 6

The simplest way is to “shim” the session’s request function:

import requests
import functools

s = requests.Session()
s.request = functools.partial(s.request, timeout=3)

# now all get, post, head etc requests should timeout after 3 seconds
# following will fail
s.get('https://httpbin.org/delay/6')

# we can still pass higher timeout when needed
# following will succeed
s.get('https://httpbin.org/delay/6', timeout=7)

Answered By: Pratyush

Is there a way to globally override requests' timeout setting?

Question:

Answers: