Is it possible to make a decorator to run blocking functions in the asyncio executor?

Question

I may may need help phrasing this question better. I’m writing an async api interface, via python3.7, & with a class (called Worker()). Worker has a few blocking methods I want to run using loop.run_in_executor().

I’d like to build a decorator I can just add above all of the non-async methods in Worker, but I keep running into problems.

I am being told that I need to await wraps() in the decorator below:

def run_method_in_executor(func, *, loop=None):
    async def wraps(*args):
        _loop = loop if loop is not None else asyncio.get_event_loop()
        return await _loop.run_in_executor(executor=None, func=func, *args)
    return wraps

which throws back:
RuntimeWarning: coroutine 'run_method_in_executor.<locals>.wraps' was never awaited

I’m not seeing how I could properly await wraps() since the containing function & decorated functions aren’t asynchronous. Not sure if this is due to misunderstanding asyncio, or misunderstanding decorators.

Any help (or help clarifying) would be greatly appreciated!

Asked By: Rob Truxal

||

Source

Answer 1

Not_a_Golfer answered my question in the comments.

Changing the inner wraps() function from a coroutine into a generator solved the problem:

def run_method_in_executor(func, *, loop=None):
    def wraps(*args):
        _loop = loop if loop is not None else asyncio.get_event_loop()
        yield _loop.run_in_executor(executor=None, func=func, *args)
    return wraps

Edit:
This has been really useful for IO, but I haven’t figured out how to await the yielded executor function, which means it will create a race condition if I’m relying on a decorated function to update some value used by any of my other async functions.

Answered By: Rob Truxal

Answer 2

Here is a complete example for Python 3.6+ which does not use interfaces deprecated by 3.8. Returning the value of loop.run_in_executor effectively converts the wrapped function to an awaitable which executes in a thread, so you can await its completion.

#!/usr/bin/env python3

import asyncio
import functools
import time


def run_in_executor(_func):
    @functools.wraps(_func)
    def wrapped(*args, **kwargs):
        loop = asyncio.get_event_loop()
        func = functools.partial(_func, *args, **kwargs)
        return loop.run_in_executor(executor=None, func=func)
    return wrapped


@run_in_executor
def say(text=None):
    """Block, then print."""
    time.sleep(1.0)
    print(f'say {text} at {time.monotonic():.3f}')


async def main():
    print(f'beginning at {time.monotonic():.3f}')
    await asyncio.gather(say('asdf'), say('hjkl'))
    await say(text='foo')
    await say(text='bar')


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

beginning at 3461039.617
say asdf at 3461040.618
say hjkl at 3461040.618
say foo at 3461041.620
say bar at 3461042.621

Answered By: Matt Vollrath

Answer 3

since the release of PEP-612 in Python 3.10 there is a way to create a decorator that also keeps the typechecker happy

from typing import Awaitable, Callable, ParamSpec, TypeVar
R = TypeVar("R")
P = ParamSpec("P")

def make_async(_func: Callable[P, R]) -> Callable[P, Awaitable[R]]:
    async def wrapped(*args: P.args, **kwargs: P.kwargs) -> R:
        func = functools.partial(_func, *args, **kwargs)
        return await asyncio.get_event_loop().run_in_executor(executor=None, func=func)

    return wrapped

Answered By: Cedric De Smet

Is it possible to make a decorator to run blocking functions in the asyncio executor?

Question:

Answers: