Python 3.5 async/await with real code example

Question

I’ve read tons of articles and tutorial about Python’s 3.5 async/await thing. I have to say I’m pretty confused, because some use get_event_loop() and run_until_complete(), some use ensure_future(), some use asyncio.wait(), and some use call_soon().

It seems like I have a lot choices, but I have no idea if they are completely identical or there are cases where you use loops and there are cases where you use wait().

But the thing is all examples work with asyncio.sleep() as simulation of real slow operation which returns an awaitable object. Once I try to swap this line for some real code the whole thing fails. What the heck are the differences between approaches written above and how should I run a third-party library which is not ready for async/await. I do use the Quandl service to fetch some stock data.

 import asyncio
 import quandl

 async def slow_operation(n):
     # await asyncio.sleep(1) # Works because it's await ready.
     await quandl.Dataset(n) # Doesn't work because it's not await ready.


 async def main():
     await asyncio.wait([
         slow_operation("SIX/US9884981013EUR4"),
         slow_operation("SIX/US88160R1014EUR4"),
     ])

 # You don't have to use any code for 50 requests/day.
 quandl.ApiConfig.api_key = "MY_SECRET_CODE"

 loop = asyncio.get_event_loop()
 loop.run_until_complete(main())

I hope you get the point how lost I feel and how simple thing I would like to have running in parallel.

Asked By: n1_

||

Source

Answer 1

If a third-party library is not compatible with async/await then obviously you can’t use it easily. There are two cases:

Let’s say that the function in the library is asynchronous and it gives you a callback, e.g.
```
def fn(..., clb):
    ...
```
So you can do:
```
def on_result(...):
    ...

fn(..., on_result)
```
In that case you can wrap such functions into the asyncio protocol like this:
```
from asyncio import Future

def wrapper(...):
    future = Future()
    def my_clb(...):
        future.set_result(xyz)
    fn(..., my_clb)
    return future
```
(use future.set_exception(exc) on exception)

Then you can simply call that wrapper in some async function with await:
```
value = await wrapper(...)
```
Note that await works with any Future object. You don’t have to declare wrapper as async.

If the function in the library is synchronous then you can run it in a separate thread (probably you would use some thread pool for that). The whole code may look like this:

import asyncio
import time
from concurrent.futures import ThreadPoolExecutor

# Initialize 10 threads
THREAD_POOL = ThreadPoolExecutor(10)

def synchronous_handler(param1, ...):
    # Do something synchronous
    time.sleep(2)
    return "foo"

# Somewhere else
async def main():
    loop = asyncio.get_event_loop()
    futures = [
        loop.run_in_executor(THREAD_POOL, synchronous_handler, param1, ...),
        loop.run_in_executor(THREAD_POOL, synchronous_handler, param1, ...),
        loop.run_in_executor(THREAD_POOL, synchronous_handler, param1, ...),
    ]
    await asyncio.wait(futures)
    for future in futures:
        print(future.result())

with THREAD_POOL:
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

If you can’t use threads for whatever reason then using such a library simply makes entire asynchronous code pointless.

Note however that using synchronous library with async is probably a bad idea. You won’t get much and yet you complicate the code a lot.

Answered By: freakish

Answer 2

You can take a look at the following simple working example from here. By the way it returns a string worth reading 🙂

import aiohttp
import asyncio

async def fetch(client):
  async with client.get('https://docs.aiohttp.org/en/stable/client_reference.html') as resp:
    assert resp.status == 200
    return await resp.text()

async def main():
  async with aiohttp.ClientSession() as client:
    html = await fetch(client)
    print(html)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

Answered By: Roman

Python 3.5 async/await with real code example

Question:

Answers: