Reusing connections in Django with Python Requests

Question

What’s the correct way of reusing Python Requests connections in Django across multiple HTTP requests. That’s what I’m doing currently:

import requests

def do_request(data):
    return requests.get('http://foo.bar/', data=data, timeout=4)

def my_view1(request)
    req = do_request({x: 1})
    ...

def my_view2(request)
    req = do_request({y: 2})
    ...

So, I have one function that makes the request. This function is called in various Django views. The views get called in separate HTTP requests by users. My question is: Does Python Requests automatically reuse the same connections (via urllib3 connection pooling)?

Or do I have to first create a Requests session object to work with?

s = requests.Session()  

def do_request(data):
    return s.get('http://foo.bar/', data=data, auth=('user', 'pass'), timeout=4).text

And if so, does the session object have to be created in global scope or should it be inside the function?

def do_request(data):
    s = requests.Session()  
    return s.get('http://foo.bar/', data=data, auth=('user', 'pass'), timeout=4).text

I can have multiple HTTP requests at the same time, so the solution needs to e thread safe … I’m a newbie to connection pooling, so I really am not sure and the Requests docs aren’t that extensive here.

Asked By: Simon Steinberger

||

Source

Answer 1

Create a session, keep the session maintained by passing it through functions and returning it, or create the session object at global level or class level, so the latest state is maintained whenever it is referenced. And it will work like a charm.

Answered By: Vikas Ojha

Answer 2

For thread-safety, do not hold a global Session object. They should be mostly safe, but there are unresolved discussions about it. See Is the Session object from Python's Requests library thread safe? and Document threading contract for Session class #2766.

Yet, for the purpose of reusing connections, it should be safe to hold a global HTTPAdapter instance, the class actually operating the underlying urllib3 PoolManager and ConnectionPool.

Mount the adapter in your sessions and maybe use the sessions from a service class for easier access and setup.

import atexit
import requests.adapters

adapter = requests.adapters.HTTPAdapter()
# Fix ResourceWarning, but at interpreter exit
# the connections are about to be closed anyway
atexit.register(adapter.close)


class FooService():
    URL = "http://foo.bar/"

    def __init__(self):
        self.session = requests.Session()
        self.session.mount("http://", adapter)  # <------ THIS

    def get_bar(self, data):
        return self.session.get(self.URL, data)


def my_view1(request)
    foo = FooService()
    foo.get_bar({"y": 1})
    # ...


def my_view2(request)
    foo = FooService()
    for y in range(1234):
        foo.get_bar({"y": y})
    # ...

Answered By: N1ngu

Reusing connections in Django with Python Requests

Question:

Answers: