asyncio with synchronous code

Question

I have a module which makes blocking network requests to some TCP server and receive responses. I must integrate it into asyncio application. My module looks like this:

import socket

# class providing transport facilities
# can be implemented in any manner
# for example it can manage asyncio connection
class CustomTransport:
    def __init__(self, host, port):
        self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.host = host
        self.port = port

    def write(self, data):
        left = len(data)
        while left:
            written = self.sock.send(data.encode('utf-8'))
            left = left - written
            data = data[written:]

    def read(self, sz):
        return self.sock.recv(sz)

    def open(self):
        self.sock.connect((self.host, self.port))

    def close(self):
        self.sock.shutdown(2)


# generated. shouldn't be modified
# however any transport can be passed
class HelloNetClient:
    def __init__(self, transport):
        self.transport = transport

    def say_hello_net(self):
        self.transport.write('hello')
        response = self.transport.read(5)
        return response


# can be modified
class HelloService:
    def __init__(self):
        # create transport for connection to echo TCP server
        self.transport = CustomTransport('127.0.0.1', 6789)
        self.hello_client = HelloNetClient(self.transport)

    def say_hello(self):
        print('Saying hello...')
        return self.hello_client.say_hello_net()

    def __enter__(self):
        self.transport.open()
        return self

    def __exit__(self,exc_type, exc_val, exc_tb):
        self.transport.close()

Usage:

def start_conversation():
    with HelloService() as hs:
        answer = hs.say_hello()
        print(answer.decode('utf-8'))


if __name__ == "__main__":
    start_conversation()

Now I see that only way to turn my module to be compatible with asyncio is to convert everything to coroutines and replace regular socket with asyncio-provided transport. But I don’t want to touch generated code (HelloNetClient). Is it possible?

P.S. I want it to be used like this:

async def start_conversation():
    async with HelloService() as hs:
        answer = await hs.say_hello()
        print(answer.decode('utf-8'))


if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(start_conversation())

Asked By: Sergey Shevchik

||

Source

Answer 1

HelloService will probably need to use run_in_executor (which manages a thread pool) to run HelloNetClient methods in the background. For example:

async def say_hello(self):
    print('Saying hello...')
    loop = asyncio.get_event_loop()
    return await loop.run_in_executor(None, self.hello_client.say_hello_net)

This is not an idiomatic use of asyncio, and you’re missing out on some of its features – for example, you won’t be able to create thousands of clients that all work in parallel, and you won’t get reliable cancellation (ability to cancel() any task you wish). Nonetheless, simple usage will work just fine.

Unfortunately the ability to provide custom transports is not of help here because the middle tier, HelloNetClient, expects synchronous behavior. Even if you were to write a custom transport that hooked into asyncio, methods like say_hello_net would still wait for as long as it takes for the response to arrive, so HelloService would have to schedule them in a separate thread. For this reason your best bet is to use the default transport and connect the code with asyncio with the code in the service as shown above.

Answered By: user4815162342

asyncio with synchronous code

Question:

Answers: