Running trivial async code from sync in Python

Question

I’m writing some parser code to be used with async IO functions (using Trio). The parser gets passed an object that exports an async read() method, and calls that method in the course of parsing.

Ordinarily, this code will be run using data straight off the network, and use Trio’s network functions. For this purpose, Trio is obviously required. However, I’d also like to be able to call the parser with a full message already in hand. In this case, the network code can be effectively replaced by a trivial async reimplementation of BytesIO or similar.

Because it awaits the implementation’s async functions, the parser code must therefore also be async. Is there an easy way to run this async code from a sync function, without running a full event loop, in the case where the read() method is guaranteed never to block?

E.g.

async def parse(reader):
    d = await reader.read(2)
    # parse data
    d2 = await reader.read(4)
    # more parsing
    return parsed_obj

Can you create an object with a never-blocking async read() method, then easily call parse() from sync code, without using an event loop?

Asked By: Tom Hunt

||

Source

Answer 1

Sure you can.

>>> async def x():
...     return 42
... 
>>> v=x()
>>> v.send(None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration: 42
>>>

Thus,

>>> def sync_call(p, *a, **kw):
...     try:
...             p(*a, **kw).send(None)
...     except StopIteration as exc:
...             return exc.value
...     raise RuntimeError("Async function yielded")
... 
>>> sync_call(x)
42
>>>

This does work across function calls, assuming that nothing in your call chain yields to a (non-existing) main loop:

>>> async def y(f):
...     return (await f())/2
... 
>>> sync_call(y,x)
21.0

Answered By: Matthias Urlichs

Answer 2

You mention there’s a concept of a "whole message" but then I’d expect there’s really two protocols on the wire: a framing protocol (e.g. size prefix or terminator character) and the message protocol. Something like this:

async def read_and_decode_message(reader, buffer):
    assert isinstance(buffer, bytearray)
    while True:
        end_of_message = find_message_end_in_buffer(buffer)
        if end_of_message is not None:
            break
        buffer.extend(await reader.read_some())
    message_encoded = buffer[:end_of_message]
    del buffer[:end_of_message]
    return decode_message(message_encoded)

If that’s the case, surely you could just call decode_message() directly, which should be a synchronous function anyway?

I’m guessing your code isn’t anything like this or you would surely have thought of that already, but maybe it could be refactored this way.

Answered By: Arthur Tacca

Running trivial async code from sync in Python

Question:

Answers: