asynchronous programming in python
Question:
Is there a generic notion of asynchronous programming in python? Could I assign a callback to a function, execute it and return to the main program flow immediately, no matter how long the execution of that function would take?
Answers:
Take a look here:
Asynchronous Programming in Python
An Introduction to Asynchronous Programming and Twisted
Worth checking out:
asyncio (previously Tulip) has been checked into the Python default branch
Edited on 14-Mar-2018
Today Python has asyncIO — Asynchronous I/O, event loop, coroutines and tasks built in.
Description taken from the link above:
The asyncIO module provides infrastructure for writing single-threaded
concurrent code using coroutines, multiplexing I/O access over sockets
and other resources, running network clients and servers, and other
related primitives. Here is a more detailed list of the package
contents:
- a pluggable event loop with various system-specific implementations;
- transport and protocol abstractions (similar to those in Twisted);
- concrete support for TCP, UDP, SSL, subprocess pipes, delayed calls,
and others (some may be system-dependent);
- a Future class that mimics the one in the concurrent.futures module, but adapted for use with the event loop;
- coroutines and tasks based on yield from (PEP 380), to
help write concurrent code in a sequential fashion;
- cancellation support for Futures and coroutines;
- synchronization primitives for use
between coroutines in a single thread, mimicking those in the
threading module;
- an interface for passing work off to a threadpool,
for times when you absolutely, positively have to use a library that
makes blocking I/O calls.
Asynchronous programming is more complex
than classical “sequential” programming: see the Develop with asyncio
page which lists common traps and explains how to avoid them. Enable
the debug mode during development to detect common issues.
Also worth checking out:
You may well want to checkout the Twisted library for Python. They provide many useful tools.
What you describe (the main program flow resuming immediately while another function executes) is not what’s normally called “asynchronous” (AKA “event-driven”) programming, but rather “multitasking” (AKA “multithreading” or “multiprocessing”). You can get what you described with the standard library modules threading and multiprocessing (the latter allows actual concurrent execution on multi-core machines).
Asynchronous (event-driven) programming is supported in the standard Python library in the asyncore and asynchat modules, which are very oriented to networking tasks (indeed they internally use the select module, which, on Windows, only supports sockets — though on Unixy OSs it can also support any file descriptor).
For a more general (though also mostly networking oriented, but not limited to that) support for asynchronous (event-driven) programming, check out the twisted third-party package.
The other respondents are pointing you to Twisted, which is a great and very comprehensive framework but in my opinion it has a very un-pythonic design. Also, AFAICT, you have to use the Twisted main loop, which may be a problem for you if you’re already using something else that provides its own loop.
Here is a contrived example that would demonstrate using the threading module:
from threading import Thread
def background_stuff():
while True:
print "I am doing some stuff"
t = Thread(target=background_stuff)
t.start()
# Continue doing some other stuff now
However, in pretty much every useful case, you will want to communicate between threads. You should look into synchronization primitives, and become familiar with the concept of concurrency and the related issues.
The threading module provides many such primitives for you to use, if you know how to use them.
You may see my Python Asynchronous Programming tool: http://www.ideawu.com/blog/2010/08/delegate-in-pythonpython-asynchronous-programming.html
import time, random, sys
from delegate import *
def proc(a):
time.sleep(random.random())
return str(a)
def proc_callback(handle, args=None):
ret = d.end(handle)
d = Delegate()
d.init(2) # number of workers
handle = d.begin(proc, '12345', proc_callback, 'test')
sys.stdin.readline()
d.free()
Good news everyone!
Python 3.4 would include brand new ambitious asynchronous programming implementation!
It is currently called tulip and already has an active following.
As described in PEP 3153: Asynchronous IO support and PEP 3156: Asynchronous IO Support Rebooted:
People who want to write asynchronous code in Python right now have a few options:
- asyncore and asynchat;
- something bespoke, most likely based on the select module;
- using a third party library, such as Twisted or gevent.
Unfortunately, each of these options has its downsides, which this PEP tries to address.
Despite having been part of the Python standard library for a long time, the asyncore module suffers from fundamental flaws following from an inflexible API that does not stand up to the expectations of a modern asynchronous networking module.
Moreover, its approach is too simplistic to provide developers with all the tools they need in order to fully exploit the potential of asynchronous networking.
The most popular solution right now used in production involves the use of third party libraries. These often provide satisfactory solutions, but there is a lack of compatibility between these libraries, which tends to make codebases very tightly coupled to the library they use.
This current lack of portability between different asynchronous IO libraries causes a lot of duplicated effort for third party library developers. A sufficiently powerful abstraction could mean that asynchronous code gets written once, but used everywhere.
Here is the brief overview of it’s abilities.
Is there a generic notion of asynchronous programming in python? Could I assign a callback to a function, execute it and return to the main program flow immediately, no matter how long the execution of that function would take?
Take a look here:
Asynchronous Programming in Python
An Introduction to Asynchronous Programming and Twisted
Worth checking out:
asyncio (previously Tulip) has been checked into the Python default branch
Edited on 14-Mar-2018
Today Python has asyncIO — Asynchronous I/O, event loop, coroutines and tasks built in.
Description taken from the link above:
The asyncIO module provides infrastructure for writing single-threaded
concurrent code using coroutines, multiplexing I/O access over sockets
and other resources, running network clients and servers, and other
related primitives. Here is a more detailed list of the package
contents:
- a pluggable event loop with various system-specific implementations;
- transport and protocol abstractions (similar to those in Twisted);
- concrete support for TCP, UDP, SSL, subprocess pipes, delayed calls,
and others (some may be system-dependent);- a Future class that mimics the one in the concurrent.futures module, but adapted for use with the event loop;
- coroutines and tasks based on yield from (PEP 380), to
help write concurrent code in a sequential fashion;- cancellation support for Futures and coroutines;
- synchronization primitives for use
between coroutines in a single thread, mimicking those in the
threading module;- an interface for passing work off to a threadpool,
for times when you absolutely, positively have to use a library that
makes blocking I/O calls.Asynchronous programming is more complex
than classical “sequential” programming: see the Develop with asyncio
page which lists common traps and explains how to avoid them. Enable
the debug mode during development to detect common issues.
Also worth checking out:
You may well want to checkout the Twisted library for Python. They provide many useful tools.
What you describe (the main program flow resuming immediately while another function executes) is not what’s normally called “asynchronous” (AKA “event-driven”) programming, but rather “multitasking” (AKA “multithreading” or “multiprocessing”). You can get what you described with the standard library modules threading and multiprocessing (the latter allows actual concurrent execution on multi-core machines).
Asynchronous (event-driven) programming is supported in the standard Python library in the asyncore and asynchat modules, which are very oriented to networking tasks (indeed they internally use the select module, which, on Windows, only supports sockets — though on Unixy OSs it can also support any file descriptor).
For a more general (though also mostly networking oriented, but not limited to that) support for asynchronous (event-driven) programming, check out the twisted third-party package.
The other respondents are pointing you to Twisted, which is a great and very comprehensive framework but in my opinion it has a very un-pythonic design. Also, AFAICT, you have to use the Twisted main loop, which may be a problem for you if you’re already using something else that provides its own loop.
Here is a contrived example that would demonstrate using the threading module:
from threading import Thread
def background_stuff():
while True:
print "I am doing some stuff"
t = Thread(target=background_stuff)
t.start()
# Continue doing some other stuff now
However, in pretty much every useful case, you will want to communicate between threads. You should look into synchronization primitives, and become familiar with the concept of concurrency and the related issues.
The threading module provides many such primitives for you to use, if you know how to use them.
You may see my Python Asynchronous Programming tool: http://www.ideawu.com/blog/2010/08/delegate-in-pythonpython-asynchronous-programming.html
import time, random, sys
from delegate import *
def proc(a):
time.sleep(random.random())
return str(a)
def proc_callback(handle, args=None):
ret = d.end(handle)
d = Delegate()
d.init(2) # number of workers
handle = d.begin(proc, '12345', proc_callback, 'test')
sys.stdin.readline()
d.free()
Good news everyone!
Python 3.4 would include brand new ambitious asynchronous programming implementation!
It is currently called tulip and already has an active following.
As described in PEP 3153: Asynchronous IO support and PEP 3156: Asynchronous IO Support Rebooted:
People who want to write asynchronous code in Python right now have a few options:
- asyncore and asynchat;
- something bespoke, most likely based on the select module;
- using a third party library, such as Twisted or gevent.
Unfortunately, each of these options has its downsides, which this PEP tries to address.
Despite having been part of the Python standard library for a long time, the asyncore module suffers from fundamental flaws following from an inflexible API that does not stand up to the expectations of a modern asynchronous networking module.
Moreover, its approach is too simplistic to provide developers with all the tools they need in order to fully exploit the potential of asynchronous networking.
The most popular solution right now used in production involves the use of third party libraries. These often provide satisfactory solutions, but there is a lack of compatibility between these libraries, which tends to make codebases very tightly coupled to the library they use.
This current lack of portability between different asynchronous IO libraries causes a lot of duplicated effort for third party library developers. A sufficiently powerful abstraction could mean that asynchronous code gets written once, but used everywhere.
Here is the brief overview of it’s abilities.