How to use asyncio for parallel tasks

Question

I was wondering how I could use asyncio to handle tasks similar to what nodeJS does. I want to run tasks at the same time without opening threads.
Example:

import asyncio
 
@asyncio.coroutine
def my_coroutine(task_name, seconds_to_sleep=3):
    print('{0} sleeping for: {1} seconds'.format(task_name, seconds_to_sleep))
    yield from asyncio.sleep(seconds_to_sleep)
    print('{0} is finished'.format(task_name))
 
 
loop = asyncio.get_event_loop()
tasks = [
    my_coroutine('task1', 4),
    my_coroutine('task2', 3),
    my_coroutine('task3', 2)]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()

will output:

task1 sleeping for: 4 seconds
task2 sleeping for: 3 seconds
task3 sleeping for: 2 seconds
task3 is finished
task2 is finished
task1 is finished

but when I try to do it with a different task it won’t work like that.

import asyncio
import timeit
 
@asyncio.coroutine
def my_coroutine(task_name):
    print('order placed for ' + task_name)
    print(timeit.timeit('1 + 3 ', number=50000000))
    print(task_name + ' done')
 
 
loop = asyncio.get_event_loop()
tasks = [
    my_coroutine('task1'),
    my_coroutine('task2'),
    my_coroutine('task3')]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()

outputs

order placed for task2
0.6677237730912453
task2 done
order placed for task1
0.6627442526498016
task1 done
order placed for task3
0.665618849882418
task3 done

Asked By: Zepol

||

Source

Answer 1

asyncio doesn’t run things in parallel. It runs one task until it awaits, then moves on to the next. The sleeps in your first example are what make the tasks yield control to each other. Your second example doesn’t await anything, so each task runs until completion before the event loop can give control to another task.

If you add something awaitable (e.g., asyncio.sleep) into your coroutine, each one will yield control and give the others a chance to run.

@asyncio.coroutine
def my_coroutine(task_name):
    print('order placed for ' + task_name)
    yield from asyncio.sleep(0)  # Another coroutine will resume here.
    print(timeit.timeit('1 + 3 ', number=50000000))
    yield from asyncio.sleep(0)  # Another coroutine will resume here.
    print(task_name + ' done')

Answered By: dirn

Answer 2

The asyncio documentation says below so asyncio tasks run concurrently but not parallelly.

asyncio is a library to write concurrent code using the async/await
syntax.

And, @asyncio.coroutine is deprecated since Python 3.7.14 and removed since Python 3.11.0 so instead, you should use async as shown below:

# @asyncio.coroutine
async def test():
    print("Test")

And for example, with this code below:

import asyncio

async def test1():
    for _ in range(0, 3):
        print("Test1")
        
async def test2():
    for _ in range(0, 3):
        print("Test2")
        
async def test3():
    for _ in range(0, 3):
        print("Test3")

async def call_tests():
    await asyncio.gather(test1(), test2(), test3())

asyncio.run(call_tests())

test1(), test2() and test3() are run serially as shown below:

Test1 # 0 second
Test1 # 0 second
Test1 # 0 second
Test2 # 0 second
Test2 # 0 second
Test2 # 0 second
Test3 # 0 second
Test3 # 0 second
Test3 # 0 second

And, if using await asyncio.sleep(1) in them as shown below:

import asyncio

async def test1():
    for _ in range(0, 3):
        print("Test1")
        await asyncio.sleep(1) # Here
        
async def test2():
    for _ in range(0, 3):
        print("Test2")
        await asyncio.sleep(1) # Here

async def test3():
    for _ in range(0, 3):
        print("Test3")
        await asyncio.sleep(1) # Here

async def call_tests():
    await asyncio.gather(test1(), test2(), test3())

asyncio.run(call_tests())

They are run alternately sleeping 1 second each time as shown below:

Test1 # 1 second
Test2 # 1 second
Test3 # 1 second
Test1 # 2 seconds
Test2 # 2 seconds
Test3 # 2 seconds
Test1 # 3 seconds
Test2 # 3 seconds
Test3 # 3 seconds

And, if using await asyncio.sleep(0) in them as shown below:

import asyncio

async def test1():
    for _ in range(0, 3):
        print("Test1")
        await asyncio.sleep(0) # Here
        
async def test2():
    for _ in range(0, 3):
        print("Test2")
        await asyncio.sleep(0) # Here

async def test3():
    for _ in range(0, 3):
        print("Test3")
        await asyncio.sleep(0) # Here

async def call_tests():
    await asyncio.gather(test1(), test2(), test3())

asyncio.run(call_tests())

They are run alternately without sleeping as shown below:

Test1 # 0 second
Test2 # 0 second
Test3 # 0 second
Test1 # 0 second
Test2 # 0 second
Test3 # 0 second
Test1 # 0 second
Test2 # 0 second
Test3 # 0 second

Answered By: Kai – Kazuya Ito

How to use asyncio for parallel tasks

Question:

Answers: