Convert Python program to C/C++ code?
Question:
is it possible to convert a Python program to C/C++?
I need to implement a couple of algorithms, and I’m not sure if the performance gap is big enough to justify all the pain I’d go through when doing it in C/C++ (which I’m not good at). I thought about writing one simple algorithm and benchmark it against such a converted solution. If that alone is significantly faster than the Python version, then I’ll have no other choice than doing it in C/C++.
Answers:
Yes. Look at Cython. It does just that: Converts Python to C for speedups.
Shed Skin is “a (restricted) Python-to-C++ compiler”.
If the C variant needs x hours less, then I’d invest that time in letting the algorithms run longer/again
“invest” isn’t the right word here.
-
Build a working implementation in Python. You’ll finish this long before you’d finish a C version.
-
Measure performance with the Python profiler. Fix any problems you find. Change data structures and algorithms as necessary to really do this properly. You’ll finish this long before you finish the first version in C.
-
If it’s still too slow, manually translate the well-designed and carefully constructed Python into C.
Because of the way hindsight works, doing the second version from existing Python (with existing unit tests, and with existing profiling data) will still be faster than trying to do the C code from scratch.
This quote is important.
Thompson’s Rule for First-Time Telescope Makers
It is faster to make a four-inch mirror and then a six-inch mirror than to make a six-inch mirror.
Bill McKeenan
Wang Institute
http://code.google.com/p/py2c/ looks like a possibility – they also mention on their site: Cython, Shedskin and RPython and confirm that they are converting Python code to pure C/C++ which is much faster than C/C++ riddled with Python API calls. Note: I haven’t tried it but I am going to..
Just came across this new tool in hacker news.
From their page – “Nuitka is a good replacement for the Python interpreter and compiles every construct that CPython 2.6, 2.7, 3.2 and 3.3 offer. It translates the Python into a C++ program that then uses “libpython” to execute in the same way as CPython does, in a very compatible way.”
I realize that an answer on a quite new solution is missing. If Numpy is used in the code, I would advice to try Pythran:
http://pythran.readthedocs.io/
For the functions I tried, Pythran gives extremely good results. The resulting functions are as fast as well written Fortran code (or only slightly slower) and a little bit faster than the (quite optimized) Cython solution.
The advantage compared to Cython is that you just have to use Pythran on the Python function optimized for Numpy, meaning that you do not have to expand the loops and add types for all variables in the loop. Pythran takes its time to analyse the code so it understands the operations on numpy.ndarray.
It is also a huge advantage compared to Numba or other projects based on just-in-time compilation for which (to my knowledge), you have to expand the loops to be really efficient. And then the code with the loops becomes very very inefficient using only CPython and Numpy…
A drawback of Pythran: no classes! But since only the functions that really need to be optimized have to be compiled, it is not very annoying.
Another point: Pythran supports well (and very easily) OpenMP parallelism. But I don’t think mpi4py is supported…
Another option – to convert to C++ besides Shed Skin – is Pythran.
To quote High Performance Python by Micha Gorelick and Ian Ozsvald:
Pythran is a Python-to-C++ compiler for a subset of Python that
includes partial numpy support. It acts a little like Numba and
Cython—you annotate a function’s arguments, and then it takes over
with further type annotation and code specialization. It takes
advantage of vectorization possibilities and of OpenMP-based
parallelization possibilities. It runs using Python 2.7 only.
One very interesting feature of Pythran is that it will attempt to
automatically spot parallelization opportunities (e.g., if you’re
using a map), and turn this into parallel code without requiring extra
effort from you. You can also specify parallel sections using pragma omp > directives; in this respect, it feels very similar to Cython’s
OpenMP support.
Behind the scenes, Pythran will take both normal Python and numpy code
and attempt to aggressively compile them into very fast C++—even
faster than the results of Cython.
You should note that this project is young, and you may encounter
bugs; you should also note that the development team are very friendly
and tend to fix bugs in a matter of hours.
I know this is an older thread but I wanted to give what I think to be helpful information.
I personally use PyPy which is really easy to install using pip. I interchangeably use Python/PyPy interpreter, you don’t need to change your code at all and I’ve found it to be roughly 40x faster than the standard python interpreter (Either Python 2x or 3x). I use pyCharm Community Edition to manage my code and I love it.
I like writing code in python as I think it lets you focus more on the task than the language, which is a huge plus for me. And if you need it to be even faster, you can always compile to a binary for Windows, Linux, or Mac (not straight forward but possible with other tools). From my experience, I get about 3.5x speedup over PyPy when compiling, meaning 140x faster than python. PyPy is available for Python 3x and 2x code and again if you use an IDE like PyCharm you can interchange between say PyPy, Cython, and Python very easily (takes a little of initial learning and setup though).
Some people may argue with me on this one, but I find PyPy to be faster than Cython. But they’re both great choices though.
Edit: I’d like to make another quick note about compiling: when you compile, the resulting binary is much bigger than your python script as it builds all dependencies into it, etc. But then you get a few distinct benefits: speed!, now the app will work on any machine (depending on which OS you compiled for, if not all. lol) without Python or libraries, it also obfuscates your code and is technically ‘production’ ready (to a degree). Some compilers also generate C code, which I haven’t really looked at or seen if it’s useful or just gibberish. Good luck.
Hope that helps.
is it possible to convert a Python program to C/C++?
I need to implement a couple of algorithms, and I’m not sure if the performance gap is big enough to justify all the pain I’d go through when doing it in C/C++ (which I’m not good at). I thought about writing one simple algorithm and benchmark it against such a converted solution. If that alone is significantly faster than the Python version, then I’ll have no other choice than doing it in C/C++.
Yes. Look at Cython. It does just that: Converts Python to C for speedups.
Shed Skin is “a (restricted) Python-to-C++ compiler”.
If the C variant needs x hours less, then I’d invest that time in letting the algorithms run longer/again
“invest” isn’t the right word here.
-
Build a working implementation in Python. You’ll finish this long before you’d finish a C version.
-
Measure performance with the Python profiler. Fix any problems you find. Change data structures and algorithms as necessary to really do this properly. You’ll finish this long before you finish the first version in C.
-
If it’s still too slow, manually translate the well-designed and carefully constructed Python into C.
Because of the way hindsight works, doing the second version from existing Python (with existing unit tests, and with existing profiling data) will still be faster than trying to do the C code from scratch.
This quote is important.
Thompson’s Rule for First-Time Telescope Makers
It is faster to make a four-inch mirror and then a six-inch mirror than to make a six-inch mirror.Bill McKeenan
Wang Institute
http://code.google.com/p/py2c/ looks like a possibility – they also mention on their site: Cython, Shedskin and RPython and confirm that they are converting Python code to pure C/C++ which is much faster than C/C++ riddled with Python API calls. Note: I haven’t tried it but I am going to..
Just came across this new tool in hacker news.
From their page – “Nuitka is a good replacement for the Python interpreter and compiles every construct that CPython 2.6, 2.7, 3.2 and 3.3 offer. It translates the Python into a C++ program that then uses “libpython” to execute in the same way as CPython does, in a very compatible way.”
I realize that an answer on a quite new solution is missing. If Numpy is used in the code, I would advice to try Pythran:
http://pythran.readthedocs.io/
For the functions I tried, Pythran gives extremely good results. The resulting functions are as fast as well written Fortran code (or only slightly slower) and a little bit faster than the (quite optimized) Cython solution.
The advantage compared to Cython is that you just have to use Pythran on the Python function optimized for Numpy, meaning that you do not have to expand the loops and add types for all variables in the loop. Pythran takes its time to analyse the code so it understands the operations on numpy.ndarray.
It is also a huge advantage compared to Numba or other projects based on just-in-time compilation for which (to my knowledge), you have to expand the loops to be really efficient. And then the code with the loops becomes very very inefficient using only CPython and Numpy…
A drawback of Pythran: no classes! But since only the functions that really need to be optimized have to be compiled, it is not very annoying.
Another point: Pythran supports well (and very easily) OpenMP parallelism. But I don’t think mpi4py is supported…
Another option – to convert to C++ besides Shed Skin – is Pythran.
To quote High Performance Python by Micha Gorelick and Ian Ozsvald:
Pythran is a Python-to-C++ compiler for a subset of Python that
includes partialnumpysupport. It acts a little like Numba and
Cython—you annotate a function’s arguments, and then it takes over
with further type annotation and code specialization. It takes
advantage of vectorization possibilities and of OpenMP-based
parallelization possibilities. It runs using Python 2.7 only.One very interesting feature of Pythran is that it will attempt to
automatically spot parallelization opportunities (e.g., if you’re
using amap), and turn this into parallel code without requiring extra
effort from you. You can also specify parallel sections usingpragma omp> directives; in this respect, it feels very similar to Cython’s
OpenMP support.Behind the scenes, Pythran will take both normal Python and numpy code
and attempt to aggressively compile them into very fast C++—even
faster than the results of Cython.You should note that this project is young, and you may encounter
bugs; you should also note that the development team are very friendly
and tend to fix bugs in a matter of hours.
I know this is an older thread but I wanted to give what I think to be helpful information.
I personally use PyPy which is really easy to install using pip. I interchangeably use Python/PyPy interpreter, you don’t need to change your code at all and I’ve found it to be roughly 40x faster than the standard python interpreter (Either Python 2x or 3x). I use pyCharm Community Edition to manage my code and I love it.
I like writing code in python as I think it lets you focus more on the task than the language, which is a huge plus for me. And if you need it to be even faster, you can always compile to a binary for Windows, Linux, or Mac (not straight forward but possible with other tools). From my experience, I get about 3.5x speedup over PyPy when compiling, meaning 140x faster than python. PyPy is available for Python 3x and 2x code and again if you use an IDE like PyCharm you can interchange between say PyPy, Cython, and Python very easily (takes a little of initial learning and setup though).
Some people may argue with me on this one, but I find PyPy to be faster than Cython. But they’re both great choices though.
Edit: I’d like to make another quick note about compiling: when you compile, the resulting binary is much bigger than your python script as it builds all dependencies into it, etc. But then you get a few distinct benefits: speed!, now the app will work on any machine (depending on which OS you compiled for, if not all. lol) without Python or libraries, it also obfuscates your code and is technically ‘production’ ready (to a degree). Some compilers also generate C code, which I haven’t really looked at or seen if it’s useful or just gibberish. Good luck.
Hope that helps.