How to find the cause of CancelledError in asyncio?
Question:
I have a big project which depends some third-party libraries, and sometimes its execution gets interrupted by a CancelledError.
To demonstrate the issue, let’s look at a small example:
import asyncio
async def main():
task = asyncio.create_task(foo())
# Cancel the task in 1 second.
loop = asyncio.get_event_loop()
loop.call_later(1.0, lambda: task.cancel())
await task
async def foo():
await asyncio.sleep(999)
if __name__ == '__main__':
asyncio.run(main())
Traceback:
Traceback (most recent call last):
File "/Users/ss/Library/Application Support/JetBrains/PyCharm2021.2/scratches/async.py", line 19, in <module>
asyncio.run(main())
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
return future.result()
concurrent.futures._base.CancelledError
As you can see, there’s no information about the place the CancelledError originates from. How do I find out the exact cause of it?
One approach that I came up with is to place a lot of try/except blocks which would catch the CancelledError and narrow down the place where it comes from. But that’s quite tedious.
Answers:
It is not possible to handle an async exception of a task if it does not call the call_exception_handler inside of it. That provides a context to the exception handler which can be customized using set_exception_handler.
If you are creating an async task you have to use try
/except
in the coroutine where possibly can occur an exception. I have demonstrated some minimal implementation to catch exceptions of async tasks using set_exception_handler
import asyncio
import logging
from asyncio import CancelledError
def async_error_handler(loop, context):
logger = logging.Logger(__name__)
logger.error(context.get("message"))
async def main():
loop = asyncio.get_running_loop()
# Set custom exception handler
loop.set_exception_handler(async_error_handler)
task = loop.create_task(foo())
# Cancel the task after 1 second
loop.call_later(1, task.cancel)
await task
async def foo():
try:
await asyncio.sleep(999)
except CancelledError:
# Catch the case when the coroutine has been canceled
loop = asyncio.get_running_loop()
# Emit an event to exception handler with custom context
loop.call_exception_handler(context={
"message": "Task has been canceled"
})
if __name__ == "__main__":
asyncio.run(main())
The context has more attributes that can also be customized. Read more about error handling here.
I’ve solved it by applyting a decorator to every async function in the project. The decorator’s job is simple – log a message when a CancelledError is raised from the function. This way we will see which functions (and more importantly, in which order) get cancelled.
Here’s the decorator code:
def log_cancellation(f):
async def wrapper(*args, **kwargs):
try:
return await f(*args, **kwargs)
except asyncio.CancelledError:
print(f"Cancelled {f}")
raise
return wrapper
In order to add this decorator everywhere I used regex. Find: (.*)(async def)
. Replace with: $1@log_cancellationn$1$2
.
Also to avoid importing log_cancellation
in every file I modified the builtins:
builtins.log_cancellation = log_cancellation
The rich
package has helped us to identify the cause of CancelledError
, without much code change required.
from rich.console import Console
console = Console()
if __name__ == "__main__":
try:
asyncio.run(main()) # replace main() with your entrypoint
except BaseException as e:
console.print_exception(show_locals=True)
I have a big project which depends some third-party libraries, and sometimes its execution gets interrupted by a CancelledError.
To demonstrate the issue, let’s look at a small example:
import asyncio
async def main():
task = asyncio.create_task(foo())
# Cancel the task in 1 second.
loop = asyncio.get_event_loop()
loop.call_later(1.0, lambda: task.cancel())
await task
async def foo():
await asyncio.sleep(999)
if __name__ == '__main__':
asyncio.run(main())
Traceback:
Traceback (most recent call last):
File "/Users/ss/Library/Application Support/JetBrains/PyCharm2021.2/scratches/async.py", line 19, in <module>
asyncio.run(main())
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
return future.result()
concurrent.futures._base.CancelledError
As you can see, there’s no information about the place the CancelledError originates from. How do I find out the exact cause of it?
One approach that I came up with is to place a lot of try/except blocks which would catch the CancelledError and narrow down the place where it comes from. But that’s quite tedious.
It is not possible to handle an async exception of a task if it does not call the call_exception_handler inside of it. That provides a context to the exception handler which can be customized using set_exception_handler.
If you are creating an async task you have to use try
/except
in the coroutine where possibly can occur an exception. I have demonstrated some minimal implementation to catch exceptions of async tasks using set_exception_handler
import asyncio
import logging
from asyncio import CancelledError
def async_error_handler(loop, context):
logger = logging.Logger(__name__)
logger.error(context.get("message"))
async def main():
loop = asyncio.get_running_loop()
# Set custom exception handler
loop.set_exception_handler(async_error_handler)
task = loop.create_task(foo())
# Cancel the task after 1 second
loop.call_later(1, task.cancel)
await task
async def foo():
try:
await asyncio.sleep(999)
except CancelledError:
# Catch the case when the coroutine has been canceled
loop = asyncio.get_running_loop()
# Emit an event to exception handler with custom context
loop.call_exception_handler(context={
"message": "Task has been canceled"
})
if __name__ == "__main__":
asyncio.run(main())
The context has more attributes that can also be customized. Read more about error handling here.
I’ve solved it by applyting a decorator to every async function in the project. The decorator’s job is simple – log a message when a CancelledError is raised from the function. This way we will see which functions (and more importantly, in which order) get cancelled.
Here’s the decorator code:
def log_cancellation(f):
async def wrapper(*args, **kwargs):
try:
return await f(*args, **kwargs)
except asyncio.CancelledError:
print(f"Cancelled {f}")
raise
return wrapper
In order to add this decorator everywhere I used regex. Find: (.*)(async def)
. Replace with: $1@log_cancellationn$1$2
.
Also to avoid importing log_cancellation
in every file I modified the builtins:
builtins.log_cancellation = log_cancellation
The rich
package has helped us to identify the cause of CancelledError
, without much code change required.
from rich.console import Console
console = Console()
if __name__ == "__main__":
try:
asyncio.run(main()) # replace main() with your entrypoint
except BaseException as e:
console.print_exception(show_locals=True)