Is a Python requests Session object shared between gevent greenlets, thread-safe (between greenlets)?

Question

Can a requests library session object be used across greenlets safely in a gevented program?

EDIT – ADDING MORE EXPLANATION:

When a greenlet yields because it has made a socket call to send the request to the server, can the same socket (inside the shared session object) be used safely by another greenlet to send its own request?

END EDIT

I attempted to test this with the code posted here – https://gist.github.com/donatello/0b399d0353cb29dc91b0 – however I got no errors or unexpected results. However, this does not validate thread safety.

In the test, I use a shared session object to make lots of requests and try to see if the object gets the requests mixed up – it is kind of naive, but I do not get any exceptions.

For convenience, I am re-pasting the code here:

client.py

import gevent
from gevent.monkey import patch_all
patch_all()

import requests
import json

s = requests.Session()

def make_request(s, d):
    r = s.post("http://127.0.0.1:5000", data=json.dumps({'value': d}))
    if r.content.strip() != str(d*2):
        print("Sent %s got %s" % (r.content, str(d*2)))
    if r.status_code != 200:
        print(r.status_code)
        print(r.content)

gevent.joinall([
    gevent.spawn(make_request, s, v)
    for v in range(300)
])

server.py

from gevent.wsgi import WSGIServer
from gevent.monkey import patch_all

patch_all()

from flask import Flask
from flask import request

import time
import json

app = Flask(__name__)

@app.route('/', methods=['POST', 'GET'])
def hello_world():
    d = json.loads(request.data)
    return str(d['value']*2)

if __name__ == '__main__':
    http_server = WSGIServer(('', 5000), app)
    http_server.serve_forever()

Exact library versions:

requirements.txt

Flask==0.10.1
Jinja2==2.7.2
MarkupSafe==0.23
Werkzeug==0.9.4
argparse==1.2.1
gevent==1.0.1
greenlet==0.4.2
gunicorn==18.0
itsdangerous==0.24
requests==2.3.0
wsgiref==0.1.2

Is there some other test that can check greenlet thread safety? The requests documentation is not too clear on this point.

Asked By: donatello

||

Source

Answer 1

The author of requests has also created a gevent-integration package: grequests. Use that instead.

It supports passing in a session with the session keyword:

import grequests

s = requests.Session()

requests = [grequests.post("http://127.0.0.1:5000", 
                           data=json.dumps({'value': d}), session=s)
            for d in range(300)]

responses = grequests.map(requests)
for r in responses:
    if r.content.strip() != str(d*2):
        print("Sent %s got %s" % (r.content, str(d*2)))
    if r.status_code != 200:
        print(r.status_code)
        print(r.content)

Note that while the session is shared, concurrent requests must use a separate connection (socket) each; HTTP 1.x can’t multiplex multiple queries over the same connection.

Answered By: Martijn Pieters

Is a Python requests Session object shared between gevent greenlets, thread-safe (between greenlets)?

Question:

Answers: