Why am I getting NotImplementedError with async and await on Windows?
Question:
I have this code:
import os
import time
import asyncio
async def run_command(*args):
"""
Example from:
http://asyncio.readthedocs.io/en/latest/subprocess.html
"""
# Create subprocess
process = await asyncio.create_subprocess_exec(
*args,
# stdout must a pipe to be accessible as process.stdout
stdout=asyncio.subprocess.PIPE)
# Wait for the subprocess to finish
stdout, stderr = await process.communicate()
# Result
result = stdout.decode().strip()
# Return stdout
return result
def run_asyncio_commands(tasks):
"""Run tasks using asyncio and return results"""
loop = asyncio.get_event_loop()
commands = asyncio.gather(*tasks) # Unpack list using *
results = loop.run_until_complete(commands)
loop.close()
return results
if __name__ == '__main__':
start = time.time()
cmds = [
['du', '-sh', '/Users/fredrik/Desktop'],
['du', '-sh', '/Users/fredrik'],
['du', '-sh', '/Users/fredrik/Pictures']
]
tasks = []
for cmd in cmds:
tasks.append(run_command(*cmd))
results = run_asyncio_commands(tasks)
print(results)
end = time.time()
print('Script ran in', str(end - start), 'seconds')
When I run the that code in Python 3.6.1 on my mac, I get this:
['780Kt/Users/fredrik/Desktop', '46Gt/Users/fredrik', '52Mt/Users/fredrik/Pictures']
Script ran in 6.405519008636475 seconds
But when I run the same script on Windows (but with the du commands substituted to something which works on Windows), also with Python 3.6.1, I get this:
Traceback (most recent call last):
File "C:UsersiruserDesktopasynciotest.py", line 66, in <module>
results = run_asyncio_commands(tasks)
File "C:UsersiruserDesktopasynciotest.py", line 41, in run_asyncio_commands
results = loop.run_until_complete(commands)
File "C:Usersfredrikcondaenvsdev_py36libasynciobase_events.py", line 466, in run_until_complete
return future.result()
File "C:UsersiruserDesktopasynciotest.py", line 16, in run_command
stdout=asyncio.subprocess.PIPE)
File "C:Usersfredrikcondaenvsdev_py36libasynciosubprocess.py", line 225, in create_subprocess_exec
stderr=stderr, **kwds)
File "C:Usersfredrikcondaenvsdev_py36libasynciobase_events.py", line 1190, in subprocess_exec
bufsize, **kwargs)
File "C:Usersfredrikcondaenvsdev_py36libasynciocoroutines.py", line 210, in coro
res = func(*args, **kw)
File "C:Usersfredrikcondaenvsdev_py36libasynciobase_events.py", line 340, in _make_subprocess_transp
ort
raise NotImplementedError
NotImplementedError
This is what I substitute the Unix commands with on Windows:
cmds = [['C:/Windows/system32/HOSTNAME.EXE']]
Windows version info:
Python 3.6.1 | packaged by conda-forge | (default, May 23 2017, 14:21:39) [MSC v.1900 64 bit (AMD64)] on win32
Windows 10 Pro, version 1703, OS build 15063.413
Answers:
Different event loops are implemented differently. Some of them have restrictions (sometimes OS-related). By default, Windows uses SelectorEventLoop and as you can see in doc:
SelectorEventLoop has the following limitations:
- SelectSelector is used to wait on socket events: it supports sockets and is limited to 512 sockets.
- loop.add_reader() and loop.add_writer() only accept socket handles (e.g. pipe file descriptors are not supported).
- Pipes are not supported, so the loop.connect_read_pipe() and loop.connect_write_pipe() methods are not implemented.
- Subprocesses are not supported, i.e. loop.subprocess_exec() and loop.subprocess_shell() methods are not implemented.
To run your code in Windows you can use alternative event loop available by default – ProactorEventLoop.
Replace line:
loop = asyncio.get_event_loop()
with this:
loop = asyncio.ProactorEventLoop()
asyncio.set_event_loop(loop)
Your code will work.
3.7.0 Python documentation handles this here:
https://docs.python.org/3/library/asyncio-platforms.html#asyncio-windows-subprocess
Set the event loop policy if you are using Windows – then your code will work.
In your startup, change the unix-specific section:
cmds = [
['du', '-sh', '/Users/fredrik/Desktop'],
['du', '-sh', '/Users/fredrik'],
['du', '-sh', '/Users/fredrik/Pictures']
]
to handle Windows & Unix:
if 'win32' in sys.platform:
# Windows specific event-loop policy & cmd
asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())
cmds = [['C:/Windows/system32/HOSTNAME.EXE']]
else:
# Unix default event-loop policy & cmds
cmds = [
['du', '-sh', '/Users/fredrik/Desktop'],
['du', '-sh', '/Users/fredrik'],
['du', '-sh', '/Users/fredrik/Pictures']
]
I tested in python and ipython, it seems the loop can be change in python, but not in ipython, and I also found the default loop is “ in jupyter and can not change neither.
Tests in python
C:UsersLiu.D.H>python
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> asyncio.get_event_loop_policy()
<asyncio.windows_events.WindowsProactorEventLoopPolicy object at 0x0000018054ACC790>
>>> asyncio.get_event_loop()
<stdin>:1: DeprecationWarning: There is no current event loop
<ProactorEventLoop running=False closed=False debug=False>
>>> asyncio.get_event_loop()
<ProactorEventLoop running=False closed=False debug=False>
>>> asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
>>> asyncio.get_event_loop_policy()
<asyncio.windows_events.WindowsSelectorEventLoopPolicy object at 0x0000018054A8E0E0>
>>> asyncio.get_event_loop()
<_WindowsSelectorEventLoop running=False closed=False debug=False>
>>> exit()
C:UsersLiu.D.H>
Tests in ipython
C:UsersLiu.D.H>ipython
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.5.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import asyncio
In [2]: asyncio.get_event_loop_policy()
Out[2]: <asyncio.windows_events.WindowsProactorEventLoopPolicy at 0x1df598b7e20>
In [3]: asyncio.get_event_loop()
<ipython-input-3-6908e23590ee>:1: DeprecationWarning: There is no current event loop
asyncio.get_event_loop()
Out[3]: <ProactorEventLoop running=False closed=False debug=False>
In [4]: asyncio.get_event_loop()
<ipython-input-4-6908e23590ee>:1: DeprecationWarning: There is no current event loop
asyncio.get_event_loop()
Out[4]: <ProactorEventLoop running=False closed=False debug=False>
In [5]: asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
In [6]: asyncio.get_event_loop_policy()
Out[6]: <asyncio.windows_events.WindowsSelectorEventLoopPolicy at 0x1df59a6c820>
In [7]: asyncio.get_event_loop()
<ipython-input-7-6908e23590ee>:1: DeprecationWarning: There is no current event loop
asyncio.get_event_loop()
Out[7]: <ProactorEventLoop running=False closed=False debug=False>
In [8]: asyncio.get_event_loop()
<ipython-input-8-6908e23590ee>:1: DeprecationWarning: There is no current event loop
asyncio.get_event_loop()
Out[8]: <ProactorEventLoop running=False closed=False debug=False>
In [9]: asyncio.set_event_loop(asyncio.SelectorEventLoop())
In [10]: asyncio.get_event_loop()
<ipython-input-10-6908e23590ee>:1: DeprecationWarning: There is no current event loop
asyncio.get_event_loop()
Out[10]: <ProactorEventLoop running=False closed=False debug=False>
In [11]:
I have this code:
import os
import time
import asyncio
async def run_command(*args):
"""
Example from:
http://asyncio.readthedocs.io/en/latest/subprocess.html
"""
# Create subprocess
process = await asyncio.create_subprocess_exec(
*args,
# stdout must a pipe to be accessible as process.stdout
stdout=asyncio.subprocess.PIPE)
# Wait for the subprocess to finish
stdout, stderr = await process.communicate()
# Result
result = stdout.decode().strip()
# Return stdout
return result
def run_asyncio_commands(tasks):
"""Run tasks using asyncio and return results"""
loop = asyncio.get_event_loop()
commands = asyncio.gather(*tasks) # Unpack list using *
results = loop.run_until_complete(commands)
loop.close()
return results
if __name__ == '__main__':
start = time.time()
cmds = [
['du', '-sh', '/Users/fredrik/Desktop'],
['du', '-sh', '/Users/fredrik'],
['du', '-sh', '/Users/fredrik/Pictures']
]
tasks = []
for cmd in cmds:
tasks.append(run_command(*cmd))
results = run_asyncio_commands(tasks)
print(results)
end = time.time()
print('Script ran in', str(end - start), 'seconds')
When I run the that code in Python 3.6.1 on my mac, I get this:
['780Kt/Users/fredrik/Desktop', '46Gt/Users/fredrik', '52Mt/Users/fredrik/Pictures']
Script ran in 6.405519008636475 seconds
But when I run the same script on Windows (but with the du commands substituted to something which works on Windows), also with Python 3.6.1, I get this:
Traceback (most recent call last):
File "C:UsersiruserDesktopasynciotest.py", line 66, in <module>
results = run_asyncio_commands(tasks)
File "C:UsersiruserDesktopasynciotest.py", line 41, in run_asyncio_commands
results = loop.run_until_complete(commands)
File "C:Usersfredrikcondaenvsdev_py36libasynciobase_events.py", line 466, in run_until_complete
return future.result()
File "C:UsersiruserDesktopasynciotest.py", line 16, in run_command
stdout=asyncio.subprocess.PIPE)
File "C:Usersfredrikcondaenvsdev_py36libasynciosubprocess.py", line 225, in create_subprocess_exec
stderr=stderr, **kwds)
File "C:Usersfredrikcondaenvsdev_py36libasynciobase_events.py", line 1190, in subprocess_exec
bufsize, **kwargs)
File "C:Usersfredrikcondaenvsdev_py36libasynciocoroutines.py", line 210, in coro
res = func(*args, **kw)
File "C:Usersfredrikcondaenvsdev_py36libasynciobase_events.py", line 340, in _make_subprocess_transp
ort
raise NotImplementedError
NotImplementedError
This is what I substitute the Unix commands with on Windows:
cmds = [['C:/Windows/system32/HOSTNAME.EXE']]
Windows version info:
Python 3.6.1 | packaged by conda-forge | (default, May 23 2017, 14:21:39) [MSC v.1900 64 bit (AMD64)] on win32
Windows 10 Pro, version 1703, OS build 15063.413
Different event loops are implemented differently. Some of them have restrictions (sometimes OS-related). By default, Windows uses SelectorEventLoop and as you can see in doc:
SelectorEventLoop has the following limitations:
- SelectSelector is used to wait on socket events: it supports sockets and is limited to 512 sockets.
- loop.add_reader() and loop.add_writer() only accept socket handles (e.g. pipe file descriptors are not supported).
- Pipes are not supported, so the loop.connect_read_pipe() and loop.connect_write_pipe() methods are not implemented.
- Subprocesses are not supported, i.e. loop.subprocess_exec() and loop.subprocess_shell() methods are not implemented.
To run your code in Windows you can use alternative event loop available by default – ProactorEventLoop.
Replace line:
loop = asyncio.get_event_loop()
with this:
loop = asyncio.ProactorEventLoop()
asyncio.set_event_loop(loop)
Your code will work.
3.7.0 Python documentation handles this here:
https://docs.python.org/3/library/asyncio-platforms.html#asyncio-windows-subprocess
Set the event loop policy if you are using Windows – then your code will work.
In your startup, change the unix-specific section:
cmds = [
['du', '-sh', '/Users/fredrik/Desktop'],
['du', '-sh', '/Users/fredrik'],
['du', '-sh', '/Users/fredrik/Pictures']
]
to handle Windows & Unix:
if 'win32' in sys.platform:
# Windows specific event-loop policy & cmd
asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())
cmds = [['C:/Windows/system32/HOSTNAME.EXE']]
else:
# Unix default event-loop policy & cmds
cmds = [
['du', '-sh', '/Users/fredrik/Desktop'],
['du', '-sh', '/Users/fredrik'],
['du', '-sh', '/Users/fredrik/Pictures']
]
I tested in python and ipython, it seems the loop can be change in python, but not in ipython, and I also found the default loop is “ in jupyter and can not change neither.
Tests in python
C:UsersLiu.D.H>python
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> asyncio.get_event_loop_policy()
<asyncio.windows_events.WindowsProactorEventLoopPolicy object at 0x0000018054ACC790>
>>> asyncio.get_event_loop()
<stdin>:1: DeprecationWarning: There is no current event loop
<ProactorEventLoop running=False closed=False debug=False>
>>> asyncio.get_event_loop()
<ProactorEventLoop running=False closed=False debug=False>
>>> asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
>>> asyncio.get_event_loop_policy()
<asyncio.windows_events.WindowsSelectorEventLoopPolicy object at 0x0000018054A8E0E0>
>>> asyncio.get_event_loop()
<_WindowsSelectorEventLoop running=False closed=False debug=False>
>>> exit()
C:UsersLiu.D.H>
Tests in ipython
C:UsersLiu.D.H>ipython
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.5.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import asyncio
In [2]: asyncio.get_event_loop_policy()
Out[2]: <asyncio.windows_events.WindowsProactorEventLoopPolicy at 0x1df598b7e20>
In [3]: asyncio.get_event_loop()
<ipython-input-3-6908e23590ee>:1: DeprecationWarning: There is no current event loop
asyncio.get_event_loop()
Out[3]: <ProactorEventLoop running=False closed=False debug=False>
In [4]: asyncio.get_event_loop()
<ipython-input-4-6908e23590ee>:1: DeprecationWarning: There is no current event loop
asyncio.get_event_loop()
Out[4]: <ProactorEventLoop running=False closed=False debug=False>
In [5]: asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
In [6]: asyncio.get_event_loop_policy()
Out[6]: <asyncio.windows_events.WindowsSelectorEventLoopPolicy at 0x1df59a6c820>
In [7]: asyncio.get_event_loop()
<ipython-input-7-6908e23590ee>:1: DeprecationWarning: There is no current event loop
asyncio.get_event_loop()
Out[7]: <ProactorEventLoop running=False closed=False debug=False>
In [8]: asyncio.get_event_loop()
<ipython-input-8-6908e23590ee>:1: DeprecationWarning: There is no current event loop
asyncio.get_event_loop()
Out[8]: <ProactorEventLoop running=False closed=False debug=False>
In [9]: asyncio.set_event_loop(asyncio.SelectorEventLoop())
In [10]: asyncio.get_event_loop()
<ipython-input-10-6908e23590ee>:1: DeprecationWarning: There is no current event loop
asyncio.get_event_loop()
Out[10]: <ProactorEventLoop running=False closed=False debug=False>
In [11]: