asyncio.Semaphore RuntimeError: Task got Future attached to a different loop
Question:
When I run this code in Python 3.7:
import asyncio
sem = asyncio.Semaphore(2)
async def work():
async with sem:
print('working')
await asyncio.sleep(1)
async def main():
await asyncio.gather(work(), work(), work())
asyncio.run(main())
It fails with RuntimeError:
$ python3 demo.py
working
working
Traceback (most recent call last):
File "demo.py", line 13, in <module>
asyncio.run(main())
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
return future.result()
File "demo.py", line 11, in main
await asyncio.gather(work(), work(), work())
File "demo.py", line 6, in work
async with sem:
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/locks.py", line 92, in __aenter__
await self.acquire()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/locks.py", line 474, in acquire
await fut
RuntimeError: Task <Task pending coro=<work() running at demo.py:6> cb=[gather.<locals>._done_callback() at /opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/tasks.py:664]> got Future <Future pending> attached to a different loop
Answers:
Python 3.10+: This error message should not occur anymore, see answer from @mmdanziger:
(…) the implementation of Semaphore has been changed and no longer grabs the current loop on init
Python 3.9 and older:
It’s because Semaphore constructor sets its _loop
attribute – in asyncio/locks.py:
class Semaphore(_ContextManagerMixin):
def __init__(self, value=1, *, loop=None):
if value < 0:
raise ValueError("Semaphore initial value must be >= 0")
self._value = value
self._waiters = collections.deque()
if loop is not None:
self._loop = loop
else:
self._loop = events.get_event_loop()
But asyncio.run()
starts a completely new loop – in asyncio/runners.py, it’s also metioned in the documentation:
def run(main, *, debug=False):
if events._get_running_loop() is not None:
raise RuntimeError(
"asyncio.run() cannot be called from a running event loop")
if not coroutines.iscoroutine(main):
raise ValueError("a coroutine was expected, got {!r}".format(main))
loop = events.new_event_loop()
...
Semaphore
initiated outside of asyncio.run()
grabs the asyncio "default" loop and so cannot be used with the event loop created with asyncio.run()
.
Solution
Initiate Semaphore
from code called by asyncio.run()
. You will have to pass them to the right place, there are more possibilities how to do that, you can for example use contextvars, but I will just give the simplest example:
import asyncio
async def work(sem):
async with sem:
print('working')
await asyncio.sleep(1)
async def main():
sem = asyncio.Semaphore(2)
await asyncio.gather(work(sem), work(sem), work(sem))
asyncio.run(main())
The same issue (and solution) is probably also with asyncio.Lock
, asyncio.Event
, and asyncio.Condition
.
Update: As of Python 3.10 OP’s code will run as written. This is because the implementation of Semaphore
has been changed and no longer grabs the current loop on init. See this answer for more discussion.
Python 3.10 implementation from GitHub
class Semaphore(_ContextManagerMixin, mixins._LoopBoundMixin):
"""A Semaphore implementation.
A semaphore manages an internal counter which is decremented by each
acquire() call and incremented by each release() call. The counter
can never go below zero; when acquire() finds that it is zero, it blocks,
waiting until some other thread calls release().
Semaphores also support the context management protocol.
The optional argument gives the initial value for the internal
counter; it defaults to 1. If the value given is less than 0,
ValueError is raised.
"""
def __init__(self, value=1, *, loop=mixins._marker):
super().__init__(loop=loop)
if value < 0:
raise ValueError("Semaphore initial value must be >= 0")
self._value = value
self._waiters = collections.deque()
self._wakeup_scheduled = False
When I run this code in Python 3.7:
import asyncio
sem = asyncio.Semaphore(2)
async def work():
async with sem:
print('working')
await asyncio.sleep(1)
async def main():
await asyncio.gather(work(), work(), work())
asyncio.run(main())
It fails with RuntimeError:
$ python3 demo.py
working
working
Traceback (most recent call last):
File "demo.py", line 13, in <module>
asyncio.run(main())
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
return future.result()
File "demo.py", line 11, in main
await asyncio.gather(work(), work(), work())
File "demo.py", line 6, in work
async with sem:
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/locks.py", line 92, in __aenter__
await self.acquire()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/locks.py", line 474, in acquire
await fut
RuntimeError: Task <Task pending coro=<work() running at demo.py:6> cb=[gather.<locals>._done_callback() at /opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/tasks.py:664]> got Future <Future pending> attached to a different loop
Python 3.10+: This error message should not occur anymore, see answer from @mmdanziger:
(…) the implementation of Semaphore has been changed and no longer grabs the current loop on init
Python 3.9 and older:
It’s because Semaphore constructor sets its _loop
attribute – in asyncio/locks.py:
class Semaphore(_ContextManagerMixin):
def __init__(self, value=1, *, loop=None):
if value < 0:
raise ValueError("Semaphore initial value must be >= 0")
self._value = value
self._waiters = collections.deque()
if loop is not None:
self._loop = loop
else:
self._loop = events.get_event_loop()
But asyncio.run()
starts a completely new loop – in asyncio/runners.py, it’s also metioned in the documentation:
def run(main, *, debug=False):
if events._get_running_loop() is not None:
raise RuntimeError(
"asyncio.run() cannot be called from a running event loop")
if not coroutines.iscoroutine(main):
raise ValueError("a coroutine was expected, got {!r}".format(main))
loop = events.new_event_loop()
...
Semaphore
initiated outside of asyncio.run()
grabs the asyncio "default" loop and so cannot be used with the event loop created with asyncio.run()
.
Solution
Initiate Semaphore
from code called by asyncio.run()
. You will have to pass them to the right place, there are more possibilities how to do that, you can for example use contextvars, but I will just give the simplest example:
import asyncio
async def work(sem):
async with sem:
print('working')
await asyncio.sleep(1)
async def main():
sem = asyncio.Semaphore(2)
await asyncio.gather(work(sem), work(sem), work(sem))
asyncio.run(main())
The same issue (and solution) is probably also with asyncio.Lock
, asyncio.Event
, and asyncio.Condition
.
Update: As of Python 3.10 OP’s code will run as written. This is because the implementation of Semaphore
has been changed and no longer grabs the current loop on init. See this answer for more discussion.
Python 3.10 implementation from GitHub
class Semaphore(_ContextManagerMixin, mixins._LoopBoundMixin):
"""A Semaphore implementation.
A semaphore manages an internal counter which is decremented by each
acquire() call and incremented by each release() call. The counter
can never go below zero; when acquire() finds that it is zero, it blocks,
waiting until some other thread calls release().
Semaphores also support the context management protocol.
The optional argument gives the initial value for the internal
counter; it defaults to 1. If the value given is less than 0,
ValueError is raised.
"""
def __init__(self, value=1, *, loop=mixins._marker):
super().__init__(loop=loop)
if value < 0:
raise ValueError("Semaphore initial value must be >= 0")
self._value = value
self._waiters = collections.deque()
self._wakeup_scheduled = False