Python/Erlang: What's the difference between Twisted, Stackless, Greenlet, Eventlet, Coroutines? Are they similar to Erlang processes?
Question:
My incomplete understanding is that Twisted, Stackless, Greenlet, Eventlet, Coroutines all make use of async network IO and userland threads that are very lightweight and quick to switch. But I’m not sure what are the differences between them.
Also they sound very similar to Erlang processes. Are they pretty much the same thing?
Anyone who could help me understand this topic more would be greatly appreciated.
Answers:
Bait taken! (fixes welcome!):
Grossly:
- twisted: single threaded. achieves non-blocking behaviour via use of ‘callbacks’ and ‘deferred’ idioms. Similar to node.js.
- greenlet / eventlet : use ‘green threads’ (sections of memory?) to achieve non-blocking io. Actually patches the standard CPython IO with their versions, so code is still written as though it is blocking / sequential.
- stackless: http://www.stackless.com/. Haven’t used it, looks like it adds ‘microthreads’ and other niceties? stackless example idioms
- coroutines: coroutines on SO
None of these are as light or well-supported as Erlang processes.
First of all, non-blocking I/O has nothing in common with green threads or coroutines, but it can affect how they’re scheduled.
Now:
- Twisted is a classic non-blocking I/O framework — application code is written in async style using callbacks.
- Gevent and eventlet use the greenlet library for coroutines/greenthreads/greenlets. There is one dedicated greenlet for running the eventloop (in case of gevent it’s C-coded libevent‘s event loop). When arbitrary greenlet begins to wait for some I/O operation to process, it just gives execution to the event loop, which starts another greenlet for execution (which is ready to do some I/O). This is called cooperative multitasking — each greenlet decides itself when to return control to other greenlets.
- Stackless has tasklets, which are similar to greenlets, but can also be scheduled with a preemptive model — that means the scheduler can stop the tasklet execution at any time and start execution of another tasklet (which is how OS threads and Erlang processes work). Also, Stackless does not provide any non-blocking I/O facilities out of the box, so if you do I/O via stdlib — it will block the entire OS thread, so no other tasklet can execute while you’re waiting on I/O. There have been attempts to provide a port of the gevent library for Stackless but I don’t know how it’s going.
You are almost right when comparing Stackless
to Greenlet. The missing thing is:
Stackless per se does not add something. Instead, Greenlet, invented 5 years after Stackless, removes certain things. It is written simple enough to be built as an extension module instead of a replacement interpreter.
This is really funny—Stackless has many more features, is about 10 times more efficient on switching, and provides pickling of execution state.
Greenlet still wins, probably only due to ease of use as an extension module. So I’m thinking about reverting the process by extending Greenlet with pickling. Maybe that would change the picture, again 🙂
My incomplete understanding is that Twisted, Stackless, Greenlet, Eventlet, Coroutines all make use of async network IO and userland threads that are very lightweight and quick to switch. But I’m not sure what are the differences between them.
Also they sound very similar to Erlang processes. Are they pretty much the same thing?
Anyone who could help me understand this topic more would be greatly appreciated.
Bait taken! (fixes welcome!):
Grossly:
- twisted: single threaded. achieves non-blocking behaviour via use of ‘callbacks’ and ‘deferred’ idioms. Similar to node.js.
- greenlet / eventlet : use ‘green threads’ (sections of memory?) to achieve non-blocking io. Actually patches the standard CPython IO with their versions, so code is still written as though it is blocking / sequential.
- stackless: http://www.stackless.com/. Haven’t used it, looks like it adds ‘microthreads’ and other niceties? stackless example idioms
- coroutines: coroutines on SO
None of these are as light or well-supported as Erlang processes.
First of all, non-blocking I/O has nothing in common with green threads or coroutines, but it can affect how they’re scheduled.
Now:
- Twisted is a classic non-blocking I/O framework — application code is written in async style using callbacks.
- Gevent and eventlet use the greenlet library for coroutines/greenthreads/greenlets. There is one dedicated greenlet for running the eventloop (in case of gevent it’s C-coded libevent‘s event loop). When arbitrary greenlet begins to wait for some I/O operation to process, it just gives execution to the event loop, which starts another greenlet for execution (which is ready to do some I/O). This is called cooperative multitasking — each greenlet decides itself when to return control to other greenlets.
- Stackless has tasklets, which are similar to greenlets, but can also be scheduled with a preemptive model — that means the scheduler can stop the tasklet execution at any time and start execution of another tasklet (which is how OS threads and Erlang processes work). Also, Stackless does not provide any non-blocking I/O facilities out of the box, so if you do I/O via stdlib — it will block the entire OS thread, so no other tasklet can execute while you’re waiting on I/O. There have been attempts to provide a port of the gevent library for Stackless but I don’t know how it’s going.
You are almost right when comparing Stackless
to Greenlet. The missing thing is:
Stackless per se does not add something. Instead, Greenlet, invented 5 years after Stackless, removes certain things. It is written simple enough to be built as an extension module instead of a replacement interpreter.
This is really funny—Stackless has many more features, is about 10 times more efficient on switching, and provides pickling of execution state.
Greenlet still wins, probably only due to ease of use as an extension module. So I’m thinking about reverting the process by extending Greenlet with pickling. Maybe that would change the picture, again 🙂