Is it possible to override the default socket options in requests?

Question

I’ve written a very simple client for a rest API using the excellent requests library for python. Everything works great util I run the client through a loadbalancer, which sanely detects idle tcp connections and kills them. I’d like for my client to use some different tcp keep alive options than are the defaults on my platform (linux). But I don’t see any easy way to tell the socket library that I’d like to choose some default options for new sockets.

When using socket.create_connection directly this is easy enough to do with a decorator, but I’ve no idea how I’d make that decorated call available when the actual call is buried in some 3rd party library as is the case with requests.

thanks in advance

Asked By: Dave Rawks

||

Source

Answer 1

requests uses urllib3, which uses the standard library’s http.client (or httplib, for 2.x), which calls socket.create_connection, all without anywhere to hook things.

So, you’re either going to have to fork one of those libraries, or monkeypatch it on the fly.

The simplest place to do it is probably in http.client.connect, since that’s a trivial wrapper around socket.create_connection that can be easily swapped out:

orig_connect = http.client.HTTPConnection.connect
def monkey_connect(self):
    orig_connect(self)
    self.sock.setsockopt(…)
http.client.HTTPConnection.connect = monkey_connect

If you’re on 2.x, it’s probably as simple as just using httplib instead of http.client above, but you may want to verify that.

Answered By: abarnert

Answer 2

Newer versions of urllib3 (since 1.8.3, released 2014-06-23) supports settings socket options.

You can set these options from requests (since 2.4.0, released 2014-08-29) by creating a custom adapter:

class HTTPAdapterWithSocketOptions(requests.adapters.HTTPAdapter):
    def __init__(self, *args, **kwargs):
        self.socket_options = kwargs.pop("socket_options", None)
        super(HTTPAdapterWithSocketOptions, self).__init__(*args, **kwargs)

    def init_poolmanager(self, *args, **kwargs):
        if self.socket_options is not None:
            kwargs["socket_options"] = self.socket_options
        super(HTTPAdapterWithSocketOptions, self).init_poolmanager(*args, **kwargs)

Then you can mount this adapter to the sessions that need custom socket options (e.g. setting SO_KEEPALIVE):

adapter = HTTPAdapterWithSocketOptions(socket_options=[(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)])
s = requests.session()
s.mount("http://", adapter)
s.mount("https://", adapter)

Answered By: univerio

Answer 3

All Browsers like FireFox, Chrome, Edge or Safari will use much frequent TCP keepalives to make sure established TCP connection remains established and they reconnect if the connection drops. On an established TCP connection there are three configurable properties that determine how keepalives work. On Linux they are:

tcp_keepalive_time (default 7200 seconds)
tcp_keepalive_probes (default 9)
tcp_keepalive_intvl (default 75 seconds)

Python requests never enables TCP keepalives on the socket (on Linux by default the TCP keepalive is not enabled on a socket, applications have to enable it). Python requests use the default socket options on each OS and hence for an HTTP 1.1 persistent connection we would not know if the established connection is dropped in case the connection remains idle. On a dropped connection we would only know when the next socket write happens. Using a lower tcp_keepalive_time than default helps diagnose a dropped idle connection. tcp_keepalive_intvl is interval between two keepalives.

In the below code we are using the requests recommended way of using an user defined HTTPAdapter to set the socket options through the underlying urllib3.
(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1) is to enable keepalives and the other two are setting the tcp_keepalive_time and tcp_keepalive_intvl to 10 seconds.

Remember TCP keepalives are platform dependent. This code is for Linux only.

import requests, socket
from requests.adapters import HTTPAdapter

class HTTPAdapterWithSocketOptions(HTTPAdapter):
    def __init__(self, *args, **kwargs):
        self.socket_options = kwargs.pop("socket_options", None)
        super(HTTPAdapterWithSocketOptions, self).__init__(*args, **kwargs)

    def init_poolmanager(self, *args, **kwargs):
        if self.socket_options is not None:
            kwargs["socket_options"] = self.socket_options
        super(HTTPAdapterWithSocketOptions, self).init_poolmanager(*args, **kwargs)

KEEPALIVE_INTERVAL = 10
adapter = HTTPAdapterWithSocketOptions(socket_options=[(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),
(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, KEEPALIVE_INTERVAL), (socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, KEEPALIVE_INTERVAL)])
s = requests.Session()
s.mount("http://", adapter)
s.mount("https://", adapter)

Answered By: Soumy

Answer 4

another available alternative is requests_toolbelt using TCPKeepAliveAdapter

which behind the seen is setting up socket of requests HTTPAdapter, and taking into account for you OSX specificities for instance.

https://toolbelt.readthedocs.io/en/latest/adapters.html#tcpkeepaliveadapter

import requests
from requests_toolbelt.adapters.socket_options import TCPKeepAliveAdapter

session = requests.Session()
keep_alive = TCPKeepAliveAdapter(idle=120, count=20, interval=30)
session.mount('https://region-a.geo-1.compute.hpcloudsvc.com', keep_alive)
session.post('https://region-a.geo-1.compute.hpcloudsvc.com/v2/1234abcdef/servers',
             # ...
           )

Answered By: ollofx

Answer 5

This can now be done via HTTPConnection.default_socket_options

import socket
from urllib3.connection import HTTPConnection

HTTPConnection.default_socket_options = ( HTTPConnection.default_socket_options + [
        (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),
        # (socket.SOL_TCP, socket.TCP_KEEPIDLE, 300),
        # (socket.SOL_TCP, socket.TCP_KEEPINTVL, 60)
        ]
)

Answered By: Always Sunny

Is it possible to override the default socket options in requests?

Question:

Answers: