How to integrate asyncronous python code into synchronous function?

Question

I have an external library that uses requests module to perform http requests.
I need to use the library asynchronously without using many threads (it would be the last choice if nothing else works). And I can’t change its source code either.
It would be easy to monkey-patch the library since all the interacting with requests module are done from a single function, but I don’t know if I can monkey-patch synchronous function with asynchronous one (I mean async keyword).

Roughly, the problem simplifies to the following code:

import asyncio
import aiohttp
import types
import requests


# Can't modify Library class.
class Library:
    def do(self):
        self._request('example.com')

        # Some other code here..

    def _request(self, url):
        return requests.get(url).text


# Monkey-patched to this method.
async def new_request(self, url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            return await response.text()


async def main():
    library = Library()

    # Do monkey-patch.
    library._request = types.MethodType(new_request, library)

    # Call library asynchronously in hope that it will perform requests using aiohttp.
    asyncio.gather(
        library.do(),
        library.do(),
        library.do()
    )

    print('Done.')

asyncio.run(main())

But as expected, it doesn’t work. I get TypeError: An asyncio.Future, a coroutine or an awaitable is required on asyncio.gather call. And also RuntimeWarning: coroutine 'new_request' was never awaited on self._request('example.com').

So the question is: is it possible to make that code work without modifying the Library class’ source code? Otherwise, what options do I have to make asynchronous requests using the library?

Asked By: g00dds

||

Source

Answer 1

Is it possible to make that code work without modifying the Library class’ source code? Otherwise, what options do I have to make asynchronous requests using the library?

Yes, it is possible, and you even do not need monkey-patching to perform that. You should use asyncio.to_thread to make the synchronous do method of Library an asynchronous function (coroutine). So the main coroutine should look like this:

async def main():
    library = Library()

    await asyncio.gather(
        asyncio.to_thread(library.do),
        asyncio.to_thread(library.do),
        asyncio.to_thread(library.do)
    )

    print('Done.')

Here the asyncio.to_thread wraps the library.do method and returns a coroutine object avoiding the first error, but you also need await before asyncio.gather.

NOTE: If you are going to check my answer with the above example, please do not forget to set a valid URL instead of ‘example.com’.

Edit

If you do not want to use threads at all, I would recommend an async wrapper like the to_async function below and replace asyncio.to_thread with that.

async def to_async(func):
    return func()


async def main():
    library = Library()

    await asyncio.gather(
        to_async(library.do),
        to_async(library.do),
        to_async(library.do),
    )

Answered By: Artyom Vancyan

Answer 2

I thought to release another answer as it solves the problem in another way.

So why do not extend the default behavior of the Library class and make the do and _request methods polymorph?

class AsyncLibrary(Library):

    async def do(self):
        return await self._request('https://google.com/')

    async def _request(self, url):
        async with aiohttp.ClientSession() as session:
            async with session.get(url) as response:
                return await response.text()


async def main():
    library = AsyncLibrary()

    await asyncio.gather(
        library.do(),
        library.do(),
        library.do(),
    )

Answered By: Artyom Vancyan

Answer 3

No way to do this with requests without threads but you can limit the number of threads active at any one time to address your without using many threads requirement.

import asyncio
import requests

# Can't modify Library class.
class Library:
    def do(self):
        self._request('http://example.com')
    def _request(self, url):
        return requests.get(url).text

async def as_thread(semaphore, func):
    async with semaphore:  # limit the number of threads active
        await asyncio.to_thread(func)

async def main():
    library = Library()
    semaphore = asyncio.Semaphore(2)  # limit to 2 for example
    tasks = [library.do] * 10  # pretend there are a lot of sites to read
    await asyncio.gather(
        *[as_thread(semaphore, x) for x in tasks]
    )
    print('Done.')

asyncio.run(main())

Answered By: jwal

How to integrate asyncronous python code into synchronous function?

Question:

Answers:

Edit