Why not always use psyco for Python code?
Question:
psyco seems to be quite helpful in optimizing Python code, and it does it in a very non-intrusive way.
Therefore, one has to wonder. Assuming you’re always on a x86 architecture (which is where most apps run these days), why not just always use psyco for all Python code? Does it make mistakes sometimes and ruins the correctness of the program? Increases the runtime for some weird cases?
Have you had any negative experiences with it? My most negative experience so far was that it made my code faster by only 15%. Usually it’s better.
Naturally, using psyco is not a replacement for efficient algorithms and coding. But if you can improve the performance of your code for the cost of two lines (importing and calling psyco), I see no good reason not to.
Answers:
When using pyglet I found that I couldn’t use psyco on the entire app without making my app non-functional. I could use it in small sections of math-heavy code, of course, but it wasn’t necessary, so I didn’t bother.
Also, psyco has done strange things with my profiling results (such as, well, not alter them at all from the non-psyco version). I suspect it doesn’t play nice with the profiling code.
I just don’t really use it unless I really want the speed, which is not all that often. My priority is algorithm optimization, which generally results in nicer speedups.
It also depends where your bottleneck is. I am mostly doing web apps and there the bottlenecks are probably more IO and database. So you should know where to optimize.
Also beware that maybe you first should think about your code instead of directly throwing psyco at it. So I agree with Devin, that algorithm optimizations should come first and they might have a smaller chance of unwanted sideeffects.
Psyco currently uses a lot of memory.
It only runs on Intel 386-compatible
processors (under any OS) right now.
There are some subtle semantic
differences (i.e. bugs) with the way
Python works; they should not be
apparent in most programs.
See also the caveats section. For a hard example, I noticed that my web app with Cheetah-generated templates and DB I/O gained no appreciable speedup.
Quite simply: “Because the code already runs fast enough”.
One should never rely on some magic bullet to fix your problems. Using psyco to make a slow program faster is usually not necessary. Bad algorithms can be rewritten, and parts that require speed could be written in another language. Of course, your question asks why we don’t use it for the speed boost anyways, and there’s a bit of overhead when you use psyco. Psyco uses memory, and those two lines just sorta feel like overhead when you look at them. As for my personal reason on why I don’t use psyco, it’s because it doesn’t support x86_64, which I see as the new up and coming architecture (especially with 2038 approaching sooner or later). My alternative is pypy, but I’m not entirely fond of that either.
A couple of other things:
- It doesn’t seem to be very actively maintained.
- It can be a memory hog.
1) The memory overhead is the main one, as described in other answers. You also pay the compilation cost, which can be prohibitive if you aren’t selective. From the user reference:
Compiling everything is often overkill for medium- or large-sized applications. The drawbacks of compiling too much are in the time spent compiling, plus the amount of memory that this process consumes. It is a subtle balance to keep.
2) Performance can actually be harmed by Psyco compilation. Again from the user guide (“known bugs” section):
There are also performance bugs: situations in which Psyco slows down the code instead of accelerating it. It is difficult to make a complete list of the possible reasons, but here are a few common ones:
- The built-in
map and filter functions must be avoided and replaced by list comprehension. For example, map(lambda x: x*x, lst) should be replaced by the more readable but more recent syntax [x*x for x in lst].
- The compilation of regular expressions doesn’t seem to benefit from Psyco. (The execution of regular expressions is unaffected, since it is C code.) Don’t enable Psyco on this module; if necessary, disable it explicitely, e.g. by calling
psyco.cannotcompile(re.compile).
3) Finally, there are some relatively obscure situations where using Psyco will actually introduce bugs. Some of them are listed here.
psyco is dead and not longer maintained. It is time to find another
psyco seems to be quite helpful in optimizing Python code, and it does it in a very non-intrusive way.
Therefore, one has to wonder. Assuming you’re always on a x86 architecture (which is where most apps run these days), why not just always use psyco for all Python code? Does it make mistakes sometimes and ruins the correctness of the program? Increases the runtime for some weird cases?
Have you had any negative experiences with it? My most negative experience so far was that it made my code faster by only 15%. Usually it’s better.
Naturally, using psyco is not a replacement for efficient algorithms and coding. But if you can improve the performance of your code for the cost of two lines (importing and calling psyco), I see no good reason not to.
When using pyglet I found that I couldn’t use psyco on the entire app without making my app non-functional. I could use it in small sections of math-heavy code, of course, but it wasn’t necessary, so I didn’t bother.
Also, psyco has done strange things with my profiling results (such as, well, not alter them at all from the non-psyco version). I suspect it doesn’t play nice with the profiling code.
I just don’t really use it unless I really want the speed, which is not all that often. My priority is algorithm optimization, which generally results in nicer speedups.
It also depends where your bottleneck is. I am mostly doing web apps and there the bottlenecks are probably more IO and database. So you should know where to optimize.
Also beware that maybe you first should think about your code instead of directly throwing psyco at it. So I agree with Devin, that algorithm optimizations should come first and they might have a smaller chance of unwanted sideeffects.
Psyco currently uses a lot of memory.
It only runs on Intel 386-compatible
processors (under any OS) right now.
There are some subtle semantic
differences (i.e. bugs) with the way
Python works; they should not be
apparent in most programs.
See also the caveats section. For a hard example, I noticed that my web app with Cheetah-generated templates and DB I/O gained no appreciable speedup.
Quite simply: “Because the code already runs fast enough”.
One should never rely on some magic bullet to fix your problems. Using psyco to make a slow program faster is usually not necessary. Bad algorithms can be rewritten, and parts that require speed could be written in another language. Of course, your question asks why we don’t use it for the speed boost anyways, and there’s a bit of overhead when you use psyco. Psyco uses memory, and those two lines just sorta feel like overhead when you look at them. As for my personal reason on why I don’t use psyco, it’s because it doesn’t support x86_64, which I see as the new up and coming architecture (especially with 2038 approaching sooner or later). My alternative is pypy, but I’m not entirely fond of that either.
A couple of other things:
- It doesn’t seem to be very actively maintained.
- It can be a memory hog.
1) The memory overhead is the main one, as described in other answers. You also pay the compilation cost, which can be prohibitive if you aren’t selective. From the user reference:
Compiling everything is often overkill for medium- or large-sized applications. The drawbacks of compiling too much are in the time spent compiling, plus the amount of memory that this process consumes. It is a subtle balance to keep.
2) Performance can actually be harmed by Psyco compilation. Again from the user guide (“known bugs” section):
There are also performance bugs: situations in which Psyco slows down the code instead of accelerating it. It is difficult to make a complete list of the possible reasons, but here are a few common ones:
- The built-in
mapandfilterfunctions must be avoided and replaced by list comprehension. For example,map(lambda x: x*x, lst)should be replaced by the more readable but more recent syntax[x*x for x in lst].- The compilation of regular expressions doesn’t seem to benefit from Psyco. (The execution of regular expressions is unaffected, since it is C code.) Don’t enable Psyco on this module; if necessary, disable it explicitely, e.g. by calling
psyco.cannotcompile(re.compile).
3) Finally, there are some relatively obscure situations where using Psyco will actually introduce bugs. Some of them are listed here.
psyco is dead and not longer maintained. It is time to find another