In what way is grequests asynchronous?

Question

I’ve been using the python requests library for some time, and recently had a need to make a request asynchronously, meaning I would like to send off the HTTP request, have my main thread continue to execute, and have a callback called when the request returns.

Naturally, I was lead to the grequests library (https://github.com/kennethreitz/grequests), but i’m confused about the behavior. For example:

import grequests

def print_res(res):
    from pprint import pprint
    pprint (vars(res))

req = grequests.get('http://www.codehenge.net/blog', hooks=dict(response=print_res))
res = grequests.map([req])

for i in range(10):
    print i

The above code will produce the following output:

<...large HTTP response output...>

0
1
2
3
4
5
6
7
8
9

The grequests.map() call obviously blocks until the HTTP response is available. It seems likely I misunderstood the ‘asynchronous’ behavior here, and the grequests library is just for performing multiple HTTP requests concurrently and sending all responses to a single callback. Is this accurate?

Asked By: cacois

||

Source

Answer 1

.map() is meant to run retrieval of several URLs in parallel, and will indeed wait for these tasks to complete (gevent.joinall(jobs)) is called).

Use .send() instead to spawn jobs, using a Pool instance:

req = grequests.get('http://www.codehenge.net/blog', hooks=dict(response=print_res))
job = grequests.send(req, grequests.Pool(1))

for i in range(10):
    print i

Without the pool the .send() call will block still, but only for the gevent.spawn() call it executes.

Answered By: Martijn Pieters

Answer 2

Create a list of requests and then send them with .imap:

event_list = [grequests.get(url_viol_card, params={"viol": i},
              session=session) for i in print_ev_list]
for r in grequests.imap(event_list, size=5):
    print(r.request.url)

session is requests.Session() object (optional)
size=5 send 5 requests simultaneously: as soon as one of them is completed the next one is sent
In this example when any request (unordered) is completed its url is printed

Answered By: Winand

Answer 3

If you don’t want to use grequests you can just implement requests with callbacks using requests + the threading module from the standard library. It’s actually really easy, and if all you want to do is send requests with callbacks the API is nicer than the one provided by grequests.

from threading import Thread

from requests import get, post, put, patch, delete, options, head



request_methods = {
    'get': get,
    'post': post,
    'put': put,
    'patch': patch,
    'delete': delete,
    'options': options,
    'head': head,
}


def async_request(method, *args, callback=None, timeout=15, **kwargs):
    """Makes request on a different thread, and optionally passes response to a
    `callback` function when request returns.
    """
    method = request_methods[method.lower()]
    if callback:
        def callback_with_args(response, *args, **kwargs):
            callback(response)
        kwargs['hooks'] = {'response': callback_with_args}
    kwargs['timeout'] = timeout
    thread = Thread(target=method, args=args, kwargs=kwargs)
    thread.start()

You can verify that it works like an AJAX calls in JS: you send a request on another thread, do some stuff on the main thread, and when the request returns you invoke a callback. This callback just prints out the response content.

async_request('get', 'http://httpbin.org/anything', callback=lambda r: print(r.json()))
for i in range(10):
    print(i)

Answered By: kylebebak

In what way is grequests asynchronous?

Question:

Answers: