What are the drawbacks of Stackless Python?
Question:
I’ve been reading recently about Stackless Python and it seems to have many advantages compared with vanilla cPython. It has all those cool features like infinite recursion, microthreads, continuations, etc. and at the same time is faster than cPython (around 10%, if the Python wiki is to be believed) and compatible with it (at least versions 2.5, 2.6 and 3.0).
All these looks almost too good to be true. However, TANSTAAFL, I don’t see much enthusiasm for Stackless among the Python community, and PEP 219 has never come into realization. Why is that? What are the drawbacks of Stackless? What skeletons are hidden in Stackless’ closet?
(I know Stackless doesn’t offer real concurrency, just an easier way of programming in the concurrent way. It doesn’t really bother me.)
Answers:
I’m also interested in the answers here. I’ve played a bit with Stackless and it looks like it would be a good solid addition to standard Python.
PEP 219 does mention potential difficulties with calling Python code from C code, if Python wants to change to a different stack. There would need to be ways to detect and prevent this (to avoid trashing the C stack). I think this is tractable though, so I’m also wondering why Stackless must stand on its own.
If I recall correctly, Stackless was slated for inclusion into the official CPython, but the author of stackless told the CPython folks not to do so, because he planned to do some significant changes to the code base – presumeably he wanted the integration done later when the project was more mature.
I don’t know where that “Stackless is 10% faster” on the Wiki came from, but then again I’ve never tried to measure those performance numbers. I can’t think of what Stackless does to make a difference that big.
Stackless is an amazing tool with several organizational/political problems.
The first comes from history. Christian Tismer started talking about what eventually became Stackless about 10 years ago. He had an idea of what he wanted, but had a hard time explaining what he was doing and why people should use it. This is partially because his background didn’t have the CS training regarding ideas like coroutines and because his presentations and discussion are very implementation oriented, which is hard for anyone not already hip-deep in continuations to understand how to use it as a solution to their problems.
For that reason, the initial documentation was poor. There were some descriptions of how to use it, with the best from third-party contributors. At PyCon 2007 I gave a talk on “Using Stackless” which went over quite well, according to the PyCon survey numbers. Richard Tew has done a great job collecting these, updating stackless.com, and maintaining the distribution when new Python releases comes up. He’s an employee of CCP Games, developers of EVE Online, which uses Stackless as an essential part of their gaming system.
CCP games is also the biggest real-world example people use when they talk about Stackless. The main tutorial for Stackless is Grant Olson’s “Introduction to Concurrent Programming with Stackless Python“, which is also game oriented. I think this gives people a skewed idea that Stackless is games-oriented, when it’s more that games are more easily continuation oriented.
Another difficulty has been the source code. In its original form it required changes to many parts of Python, which made Guido van Rossum, the Python lead, wary. Part of the reason, I think, was support for call/cc that was later removed as being “too much like supporting a goto when there are better higher-level forms.” I’m not certain about this history, so just read this paragraph as “Stackless used to require too many changes.”
Later releases didn’t require the changes, and Tismer continued to push for its inclusion in Python. While there was some consideration, the official stance (as far as I know) is that CPython is not only a Python implementation but it’s meant as a reference implementation, and it won’t include Stackless functionality because it can’t be implemented by Jython or Iron Python.
There are absolutely no plans for “significant changes to the code base“. That quote and reference hyperlink from Arafangion’s (see the comment) are from roughly 2000/2001. The structural changes have long been done, and it’s what I mentioned above. Stackless as it is now is stable and mature, with only minor tweaks to the code base over the last several years.
One final limitation with Stackless – there is no strong advocate for Stackless. Tismer is now deeply involved with PyPy, which is an implementation of Python for Python. He has implemented the Stackless functionality in PyPy and considers it much superior to Stackless itself, and feels that PyPy is the way of the future. Tew maintains Stackless but he isn’t interested in advocacy. I considered being in that role, but couldn’t see how I could make an income from it.
Though if you want training in Stackless, feel free to contact me! 🙂
it took quite long to find this discussion. At that
time I was not on PyPy but had a 2-years affair with psyco, until health stopped this all quite abruptly. I’m now active again and designing an
alternative approach – will present it on EuroPython 2012.
Most of Andrews statements are correct. Some
minor additions:
Stackless was significantly faster than CPython, 10 years ago, because I optimized the interpreter loop. At that time, Guido was not ready for that. A few years later, people did similar optimizations and even more and better ones, which makes Stackless a little bit slower, as expected.
On inclusion: well, in the beginning I was very pushy and convinced that Stackless is the way to go. Later, when it was almost possible to get included, I lost interest in that and preferred to let
it stay this way, partially out of frustration, partially to
keep control of Stackless.
The arguments like “other implementations cannot do it” felt always lame to me, as there are other examples where this argument could also be used. I thought I better forget about that and stay in good friendship with Guido, having my own distro.
Meanwhile things are changing again. I’m working on PyPy and Stackless as an extension Will talk about that sometimes later
Cheers — Chris
I’ve been reading recently about Stackless Python and it seems to have many advantages compared with vanilla cPython. It has all those cool features like infinite recursion, microthreads, continuations, etc. and at the same time is faster than cPython (around 10%, if the Python wiki is to be believed) and compatible with it (at least versions 2.5, 2.6 and 3.0).
All these looks almost too good to be true. However, TANSTAAFL, I don’t see much enthusiasm for Stackless among the Python community, and PEP 219 has never come into realization. Why is that? What are the drawbacks of Stackless? What skeletons are hidden in Stackless’ closet?
(I know Stackless doesn’t offer real concurrency, just an easier way of programming in the concurrent way. It doesn’t really bother me.)
I’m also interested in the answers here. I’ve played a bit with Stackless and it looks like it would be a good solid addition to standard Python.
PEP 219 does mention potential difficulties with calling Python code from C code, if Python wants to change to a different stack. There would need to be ways to detect and prevent this (to avoid trashing the C stack). I think this is tractable though, so I’m also wondering why Stackless must stand on its own.
If I recall correctly, Stackless was slated for inclusion into the official CPython, but the author of stackless told the CPython folks not to do so, because he planned to do some significant changes to the code base – presumeably he wanted the integration done later when the project was more mature.
I don’t know where that “Stackless is 10% faster” on the Wiki came from, but then again I’ve never tried to measure those performance numbers. I can’t think of what Stackless does to make a difference that big.
Stackless is an amazing tool with several organizational/political problems.
The first comes from history. Christian Tismer started talking about what eventually became Stackless about 10 years ago. He had an idea of what he wanted, but had a hard time explaining what he was doing and why people should use it. This is partially because his background didn’t have the CS training regarding ideas like coroutines and because his presentations and discussion are very implementation oriented, which is hard for anyone not already hip-deep in continuations to understand how to use it as a solution to their problems.
For that reason, the initial documentation was poor. There were some descriptions of how to use it, with the best from third-party contributors. At PyCon 2007 I gave a talk on “Using Stackless” which went over quite well, according to the PyCon survey numbers. Richard Tew has done a great job collecting these, updating stackless.com, and maintaining the distribution when new Python releases comes up. He’s an employee of CCP Games, developers of EVE Online, which uses Stackless as an essential part of their gaming system.
CCP games is also the biggest real-world example people use when they talk about Stackless. The main tutorial for Stackless is Grant Olson’s “Introduction to Concurrent Programming with Stackless Python“, which is also game oriented. I think this gives people a skewed idea that Stackless is games-oriented, when it’s more that games are more easily continuation oriented.
Another difficulty has been the source code. In its original form it required changes to many parts of Python, which made Guido van Rossum, the Python lead, wary. Part of the reason, I think, was support for call/cc that was later removed as being “too much like supporting a goto when there are better higher-level forms.” I’m not certain about this history, so just read this paragraph as “Stackless used to require too many changes.”
Later releases didn’t require the changes, and Tismer continued to push for its inclusion in Python. While there was some consideration, the official stance (as far as I know) is that CPython is not only a Python implementation but it’s meant as a reference implementation, and it won’t include Stackless functionality because it can’t be implemented by Jython or Iron Python.
There are absolutely no plans for “significant changes to the code base“. That quote and reference hyperlink from Arafangion’s (see the comment) are from roughly 2000/2001. The structural changes have long been done, and it’s what I mentioned above. Stackless as it is now is stable and mature, with only minor tweaks to the code base over the last several years.
One final limitation with Stackless – there is no strong advocate for Stackless. Tismer is now deeply involved with PyPy, which is an implementation of Python for Python. He has implemented the Stackless functionality in PyPy and considers it much superior to Stackless itself, and feels that PyPy is the way of the future. Tew maintains Stackless but he isn’t interested in advocacy. I considered being in that role, but couldn’t see how I could make an income from it.
Though if you want training in Stackless, feel free to contact me! 🙂
it took quite long to find this discussion. At that
time I was not on PyPy but had a 2-years affair with psyco, until health stopped this all quite abruptly. I’m now active again and designing an
alternative approach – will present it on EuroPython 2012.
Most of Andrews statements are correct. Some
minor additions:
Stackless was significantly faster than CPython, 10 years ago, because I optimized the interpreter loop. At that time, Guido was not ready for that. A few years later, people did similar optimizations and even more and better ones, which makes Stackless a little bit slower, as expected.
On inclusion: well, in the beginning I was very pushy and convinced that Stackless is the way to go. Later, when it was almost possible to get included, I lost interest in that and preferred to let
it stay this way, partially out of frustration, partially to
keep control of Stackless.
The arguments like “other implementations cannot do it” felt always lame to me, as there are other examples where this argument could also be used. I thought I better forget about that and stay in good friendship with Guido, having my own distro.
Meanwhile things are changing again. I’m working on PyPy and Stackless as an extension Will talk about that sometimes later
Cheers — Chris