Cython Speed Boost vs. Usability
Question:
I just came across Cython, while I was looking out for ways to optimize Python code. I read various posts on Stack Overflow, the python wiki and read the article "General Rules for Optimization".
Cython is something which grasps my interest the most; instead of writing C-code for yourself, you can choose to have other data types in your python code itself.
Here is a silly test I tried,
#!/usr/bin/python
# test.pyx
def test(value):
for i in xrange(value):
i**2
if(i==1000000):
print i
test(10000001)
$ time python test.pyx
real 0m16.774s
user 0m16.745s
sys 0m0.024s
$ time cython test.pyx
real 0m0.513s
user 0m0.196s
sys 0m0.052s
Now, honestly, I’m dumbfounded. The code which I have used here is pure python code, and all I have changed is the interpreter. In this case, if the cython is this good, then why do people still use the traditional Python interpreter? Are there any reliability issues for Cython?
Answers:
Cython is not another interpreter. It generates c-extensions for python, from python(-like) code. cython test.pyx will only generate a ‘test.c’ file, which (once compiled) can be used by python just like a normal python library.
That means that you are only measuring the time it takes for cython to translate your python code to c, not how fast that version of your code runs.
-
cython test.pyx doesn’t actually run your program. The cython binary is for processing your Cython code into a Python extension module. You would have to import it in Python to run it.
-
#!/usr/bin/python isn’t the best shebang line for Python scripts. #!/usr/bin/env python is generally preferred, which runs whatever python would on the command line.
- Cython
pyx files probably shouldn’t have a shebang line at all, except in the corner case they are valid Python programs.
-
You have an IndentationError in the posted code.
-
Using the traditional interpreter is simpler and more portable. Cython is reliable, but has its limitations and quirks. It might be compelling to use it tons more if it magically gave the speedups your timings make it look like it does, but it actually gives smaller ones. You’ll have to start using Cython-specific features to use C features to see a lot of speedup.
The other answers have already explained how you were just compiling the Cython code, not executing it. However, I thought that you might want to know how much faster Cython can make your code. When I compiled the code you have (though I ran the function from from a different module) with distutils, I got very marginal speed gains over straight Python – about 1%. However, when I added a few small changes to your code:
def test(long long value):
cdef long long i
cdef long long z
for i in xrange(value):
z = i**2
if(i==1000000):
print i
if z < i:
print "yes"
and compiled it, I got the following times:
- Pure Python code: 20.4553578737 seconds
- Cython code: 0.199339860234 seconds
That’s a 100× speed-up. Not too shabby.
A big point that seems to be missing: Cython is not a strict superset of Python. There are some features that Python supports, but Cython does not. Most notably, generators and lambdas (but they are coming).
-
Biggest reason Cython is not that popular is that it lacks standalone (without python) executables.
-
Lack of publicity. The developers seem to be academics more interested in Developing their Sage software, than a cutting edge language.
-
Pitfalls encountered during development. One I encountered is lack of true thread support. Everything is wrapped up in a global interpreter lock, making it threadsafe, but disabling concurrency!
For anyone who wishes that cython actually compiled and ran your program in one line, I created runcython (http://github.com/russell91/runcython). runcython test.pyx will have the semantics that the OP intended
As many of comments suggesting the time you have mentioned with cython is just time taken in converting .py file into .c file. That .c file will be big interms of memory. The advantage is that you can compile that .c file using C compiler also
I will give you one example :
if you have a python file we need to store as .pyx file. By running “cython xyz.pyx” it will create .c and .so files
Lets check in picture files created by cython command
The size also increases in my case it changed for Bytes to Kb
There is some specific method to compile the python code
you can refer this links:
http://cython.readthedocs.io/en/latest/src/reference/compilation.html
https://rajeshrinet.github.io/blog/2014/cython/
another thing i noticed is no where they mention the memory problems in cython
When i try to implement in my embedded project.I am facing lot of memory problems.
You can try pypy Where there is no need to put extra efforts to compile and the time also reduced in terms of compilation time
Refer the picture:
compare time between python and pypy
Thank you..
I just came across Cython, while I was looking out for ways to optimize Python code. I read various posts on Stack Overflow, the python wiki and read the article "General Rules for Optimization".
Cython is something which grasps my interest the most; instead of writing C-code for yourself, you can choose to have other data types in your python code itself.
Here is a silly test I tried,
#!/usr/bin/python
# test.pyx
def test(value):
for i in xrange(value):
i**2
if(i==1000000):
print i
test(10000001)
$ time python test.pyx
real 0m16.774s
user 0m16.745s
sys 0m0.024s
$ time cython test.pyx
real 0m0.513s
user 0m0.196s
sys 0m0.052s
Now, honestly, I’m dumbfounded. The code which I have used here is pure python code, and all I have changed is the interpreter. In this case, if the cython is this good, then why do people still use the traditional Python interpreter? Are there any reliability issues for Cython?
Cython is not another interpreter. It generates c-extensions for python, from python(-like) code. cython test.pyx will only generate a ‘test.c’ file, which (once compiled) can be used by python just like a normal python library.
That means that you are only measuring the time it takes for cython to translate your python code to c, not how fast that version of your code runs.
-
cython test.pyxdoesn’t actually run your program. Thecythonbinary is for processing your Cython code into a Python extension module. You would have to import it in Python to run it. -
#!/usr/bin/pythonisn’t the best shebang line for Python scripts.#!/usr/bin/env pythonis generally preferred, which runs whateverpythonwould on the command line.- Cython
pyxfiles probably shouldn’t have a shebang line at all, except in the corner case they are valid Python programs.
- Cython
-
You have an IndentationError in the posted code.
-
Using the traditional interpreter is simpler and more portable. Cython is reliable, but has its limitations and quirks. It might be compelling to use it tons more if it magically gave the speedups your timings make it look like it does, but it actually gives smaller ones. You’ll have to start using Cython-specific features to use C features to see a lot of speedup.
The other answers have already explained how you were just compiling the Cython code, not executing it. However, I thought that you might want to know how much faster Cython can make your code. When I compiled the code you have (though I ran the function from from a different module) with distutils, I got very marginal speed gains over straight Python – about 1%. However, when I added a few small changes to your code:
def test(long long value):
cdef long long i
cdef long long z
for i in xrange(value):
z = i**2
if(i==1000000):
print i
if z < i:
print "yes"
and compiled it, I got the following times:
- Pure Python code: 20.4553578737 seconds
- Cython code: 0.199339860234 seconds
That’s a 100× speed-up. Not too shabby.
A big point that seems to be missing: Cython is not a strict superset of Python. There are some features that Python supports, but Cython does not. Most notably, generators and lambdas (but they are coming).
-
Biggest reason Cython is not that popular is that it lacks standalone (without python) executables.
-
Lack of publicity. The developers seem to be academics more interested in Developing their Sage software, than a cutting edge language.
-
Pitfalls encountered during development. One I encountered is lack of true thread support. Everything is wrapped up in a global interpreter lock, making it threadsafe, but disabling concurrency!
For anyone who wishes that cython actually compiled and ran your program in one line, I created runcython (http://github.com/russell91/runcython). runcython test.pyx will have the semantics that the OP intended
As many of comments suggesting the time you have mentioned with cython is just time taken in converting .py file into .c file. That .c file will be big interms of memory. The advantage is that you can compile that .c file using C compiler also
I will give you one example :
if you have a python file we need to store as .pyx file. By running “cython xyz.pyx” it will create .c and .so files
Lets check in picture files created by cython command
The size also increases in my case it changed for Bytes to Kb
There is some specific method to compile the python code
you can refer this links:
http://cython.readthedocs.io/en/latest/src/reference/compilation.html
https://rajeshrinet.github.io/blog/2014/cython/
another thing i noticed is no where they mention the memory problems in cython
When i try to implement in my embedded project.I am facing lot of memory problems.
You can try pypy Where there is no need to put extra efforts to compile and the time also reduced in terms of compilation time
Refer the picture:
compare time between python and pypy
Thank you..