Python asyncio: handling exceptions in gather() – documentation unclear?

Question

The documentation for asyncio.gather says that

If return_exceptions is False (default), the first raised exception is
immediately propagated to the task that awaits on gather(). Other
awaitables in the aws sequence won’t be cancelled and will continue to
run.

However, from a simple test it seems that if one of the tasks raises an exception when return_exceptions is False, all other awaitable are cancelled (or to be more precise, in case the terminology is not clear to me, the other awaitables do not finish their job):

import asyncio

async def factorial(name, number, raise_exception=False):
    # If raise_exception is True, will raise an exception when
    # the loop counter > 3
    f = 1
    for i in range(2, number + 1):
        print(f'  Task {name}: Compute factorial({i})...')

        if raise_exception and i > 3:
            print(f'  Task {name}: raising Exception')
            raise Exception(f'Bad Task {name}')

        await asyncio.sleep(1)
        f *= i
    print(f'==>> Task {name} DONE: factorial({number}) = {f}')
    return f

async def main():
    tasks = [factorial('A', 5),  # this will not be finished
             factorial('B', 10, raise_exception=True),
             factorial('C', 2)]

    try:
        results = await asyncio.gather(*tasks)
        print('Results:', results)
    except Exception as e:
        print('Got an exception:', e)

asyncio.run(main())

What this piece of code is doing, just to make it simpler, it defines 3 tasks and call asyncio.gather() on them. One of the tasks raises an exception before one of the others is done, and this other task is not finished.

Actually, I cannot even make sense with what the documentations says – if an exception is raised and caught by the task awaiting on gather, I would not even be able to get the returned results (even if the other task would, somehow, get done).

Am I missing anything, or is there a problem with the documentation?

This was tested with Python 3.7.2.

Asked By: amitr

||

Source

Answer 1

I’ve run your code and got the following output, as expected from documentation.

  Task C: Compute factorial(2)...
  Task A: Compute factorial(2)...
  Task B: Compute factorial(2)...
==>> Task C DONE: factorial(2) = 2
  Task A: Compute factorial(3)...
  Task B: Compute factorial(3)...
  Task A: Compute factorial(4)...
  Task B: Compute factorial(4)...
  Task B: raising Exception
Got an exception: Bad Task B
  Task A: Compute factorial(5)...
==>> Task A DONE: factorial(5) = 120

What’s going on

Tasks A,B and C are submitted to the queue;
All tasks are running while C finishes earliest.
Task B raises and exception.
The await asyncio.gater() returns immediately and print('Got an exception:', e) to the screen.
Task A continues to run and print “==>> Task A DONE …”

What’s wrong with your test

As @deceze commented,
your program exited immediately after the exception was caught and main() returns. Thus, the tasks A and C are terminated because the entire process dies, not because of cancellation.

To fix it, add await asyncio.sleep(20) to the end of the main() function.

Answered By: gdlmx

Answer 2

The answer to the main question here is to use asyncio.as_complete. Change your main() function code to:

async def main():
    tasks = [factorial('A', 5),  # this will not be finished
             factorial('B', 10, raise_exception=True),
             factorial('C', 2)]


    # Handle results in the order the task are completed
    # if exeption you can handle that as well. 
    for coroutine in asyncio.as_completed(tasks):
        try:
            results = await coroutine
        except Exception as e:
            print('Got an exception:', e)
        else:
            print('Results:', results)

Answered By: Raydel Miranda

Python asyncio: handling exceptions in gather() – documentation unclear?

Question:

Answers:

What’s going on

What’s wrong with your test