Using Cython with Django. Does it make sense?
Question:
Is it possible to optimize speed of a mission critical application developed in Django with Cython?
Recently I have read on the internet, that Cython can turn a Python code to C like speed. Is this possible with Django?
Answers:
Well, yes, but most things a web app does won’t really benefit from this sort of change unless you have firm proof that it will. Profile twice, optimize once.
Is it possible to optimize speed of a mission critical application developed in Django with Cython
It’s doubtful.
Most of a web application response time is the non-HTML elements that must be downloaded separately. The usual rule of thumb is 8 static files per HTML page. (.CSS, .JS, images, etc.)
Since none of that static content comes from Django, most of your web application’s time-line is Apache (or Nginx or some other server software outside Django).
When looking at just the time to produce the HTML, you’ll find that most of the time is spent waiting for the database (even if it’s in-memory SQLite, you’ll see that the database tends to dominate the timeline)
When you’re through making Apache and the database go fast, then — and only then — you can consider the Python elements.
Bottom Line. Don’t waste any of your time on making Django and Python go faster.
Echoing the other answers, this is unlikely to gain you any specific gains, unless of course, you’ve profiled a specific case where some benefit could be derived.
Adding to the answers, I want to point out a list of web optimizations from the Yahoo’s web team. These are measured and proven gains in areas applicable to many websites and worth studying:
The Exceptional Performance team has identified a number of best practices for making web pages fast. The list includes 35 best practices divided into 7 categories.
Conceptually what you are after is possible, but you are looking in the wrong direction.
You really should be interested in PyPy, which is an extremely promising technology for the Django (and Python) community. Benchmarks of Django on PyPy already show a 12.5X speed gain when compared to normal Python. Now even though front end static resources may contribute to the majority of page responsiveness to the end user (per S.Lott’s answer above), this does not address the reality that speed gains on the server side are extremely important. Facebook had to invent HipHop PHP for this reason…the lag of PHP just couldn’t cut it. Closer to home, simply consider that Meebo (a python driven site) spends tens of thousands USD per month on server infrastructure. Now can you taste the server cost savings with PyPy? Sexy!
Check out the benchmarks: http://speed.pypy.org/
We have built an application that is installed on the customer site called InProd, it manages the configuration of Genesys powered contact centers. It is a Django application compiled with Cython.
There is a speed improvement but it is not considerable. We need to process a lot of requests per second and the main delay in doing this is caused by the database. The biggest gains are to be made with database tuning and good sql.
You need to have a really strong need to do such a thing as it does cause on going issues, it increases our testing requirements.
The performance difference can be large on some tasks, but not all. So the improvement will depend on what your application is doing.
It depends if you have heavy processes on the backend side. In my case it could improve one of the processes and speed it up ~5 times. I had a function that rework large XML files that is inputed by the user and save the output into database. telling Cython that in some places the input or output is string did the 5x speed up magic.
Is it possible to optimize speed of a mission critical application developed in Django with Cython?
Recently I have read on the internet, that Cython can turn a Python code to C like speed. Is this possible with Django?
Well, yes, but most things a web app does won’t really benefit from this sort of change unless you have firm proof that it will. Profile twice, optimize once.
Is it possible to optimize speed of a mission critical application developed in Django with Cython
It’s doubtful.
Most of a web application response time is the non-HTML elements that must be downloaded separately. The usual rule of thumb is 8 static files per HTML page. (.CSS, .JS, images, etc.)
Since none of that static content comes from Django, most of your web application’s time-line is Apache (or Nginx or some other server software outside Django).
When looking at just the time to produce the HTML, you’ll find that most of the time is spent waiting for the database (even if it’s in-memory SQLite, you’ll see that the database tends to dominate the timeline)
When you’re through making Apache and the database go fast, then — and only then — you can consider the Python elements.
Bottom Line. Don’t waste any of your time on making Django and Python go faster.
Echoing the other answers, this is unlikely to gain you any specific gains, unless of course, you’ve profiled a specific case where some benefit could be derived.
Adding to the answers, I want to point out a list of web optimizations from the Yahoo’s web team. These are measured and proven gains in areas applicable to many websites and worth studying:
The Exceptional Performance team has identified a number of best practices for making web pages fast. The list includes 35 best practices divided into 7 categories.
Conceptually what you are after is possible, but you are looking in the wrong direction.
You really should be interested in PyPy, which is an extremely promising technology for the Django (and Python) community. Benchmarks of Django on PyPy already show a 12.5X speed gain when compared to normal Python. Now even though front end static resources may contribute to the majority of page responsiveness to the end user (per S.Lott’s answer above), this does not address the reality that speed gains on the server side are extremely important. Facebook had to invent HipHop PHP for this reason…the lag of PHP just couldn’t cut it. Closer to home, simply consider that Meebo (a python driven site) spends tens of thousands USD per month on server infrastructure. Now can you taste the server cost savings with PyPy? Sexy!
Check out the benchmarks: http://speed.pypy.org/
We have built an application that is installed on the customer site called InProd, it manages the configuration of Genesys powered contact centers. It is a Django application compiled with Cython.
There is a speed improvement but it is not considerable. We need to process a lot of requests per second and the main delay in doing this is caused by the database. The biggest gains are to be made with database tuning and good sql.
You need to have a really strong need to do such a thing as it does cause on going issues, it increases our testing requirements.
The performance difference can be large on some tasks, but not all. So the improvement will depend on what your application is doing.
It depends if you have heavy processes on the backend side. In my case it could improve one of the processes and speed it up ~5 times. I had a function that rework large XML files that is inputed by the user and save the output into database. telling Cython that in some places the input or output is string did the 5x speed up magic.