Implementing a function in Python vs C
Question:
Is there a difference (in terms of execution time) between implementing a function in Python and implementing it in C and then calling it from Python? If so, why?
Answers:
Python (at least the “standard” CPython implementation) never actually compiles to native machine code; it compiles to bytecode which is then interpreted. So a C function which is in fact compiled to machine code will run faster; the question is whether it will make a relevant difference. So what’s the actual problem you’re trying to solve?
Typically, a function written in C will be substantially faster that the Python equivalent. It is also much more difficult to integrate, since it involves:
- compiling C code that #includes the Python headers and exposes appropriate wrapper code so that it is callable from Python;
- linking against the correct Python libraries;
- deploying the resulting shared library to the appropriate location, so that your Python code can import it.
You would want to be very certain that the benefits outweigh the costs before trying this, which means this should only be reserved for performance-critical sections of your code that you simply can’t make fast enough with pure Python.
If you really need to go down this path, Boost.Python can make the task much less painful.
If I understand and restate your question properly, you are asking, if wrapping python over a c executable be anyway faster than a pure python module itself? The answer to it is, it depends upon the executable and the kind of task you are performing.
- There are a set of modules in Python that are written using Python C-API’s. Performance of those would be comparable to wrapping a C executable
- On the other hand, wrapping c program would be faster than pure python both implementing the same functionality with a sane logic. Compare difflib usage vs wrapping subprocess over diff.
The C version is often faster, but not always. One of the main points of speedup is that C code does not have to look up values dynamically, like Python (Python has reference semantics). A good example for this is Numpy. Numpy arrays are typed, all values in the array have the same type, and are internally stored in continuous block of memory. This is the main reason that numpy is so much faster, because it skips all the dynamic variable lookup that Python has to do. The most efficient C implementation of an algorithm can become very slow if it operates on Python data structures, where each value has to be looked up dynamically.
A good way to implement such things yourself and save all the hassle of Python C-APIs is to use Cython.
Is there a difference (in terms of execution time) between implementing a function in Python and implementing it in C and then calling it from Python? If so, why?
Python (at least the “standard” CPython implementation) never actually compiles to native machine code; it compiles to bytecode which is then interpreted. So a C function which is in fact compiled to machine code will run faster; the question is whether it will make a relevant difference. So what’s the actual problem you’re trying to solve?
Typically, a function written in C will be substantially faster that the Python equivalent. It is also much more difficult to integrate, since it involves:
- compiling C code that #includes the Python headers and exposes appropriate wrapper code so that it is callable from Python;
- linking against the correct Python libraries;
- deploying the resulting shared library to the appropriate location, so that your Python code can import it.
You would want to be very certain that the benefits outweigh the costs before trying this, which means this should only be reserved for performance-critical sections of your code that you simply can’t make fast enough with pure Python.
If you really need to go down this path, Boost.Python can make the task much less painful.
If I understand and restate your question properly, you are asking, if wrapping python over a c executable be anyway faster than a pure python module itself? The answer to it is, it depends upon the executable and the kind of task you are performing.
- There are a set of modules in Python that are written using Python C-API’s. Performance of those would be comparable to wrapping a C executable
- On the other hand, wrapping c program would be faster than pure python both implementing the same functionality with a sane logic. Compare difflib usage vs wrapping subprocess over diff.
The C version is often faster, but not always. One of the main points of speedup is that C code does not have to look up values dynamically, like Python (Python has reference semantics). A good example for this is Numpy. Numpy arrays are typed, all values in the array have the same type, and are internally stored in continuous block of memory. This is the main reason that numpy is so much faster, because it skips all the dynamic variable lookup that Python has to do. The most efficient C implementation of an algorithm can become very slow if it operates on Python data structures, where each value has to be looked up dynamically.
A good way to implement such things yourself and save all the hassle of Python C-APIs is to use Cython.