Python Comet Server
Question:
I am building a web application that has a real-time feed (similar to Facebook’s newsfeed) that I want to update via a long-polling mechanism. I understand that with Python, my choices are pretty much to either use Stackless (building from their Comet wsgi example) or Cometd + Twisted. Unfortunately there is very little documentation regarding these options and I cannot find good information online about production scale users of comet on Python.
Has anyone successfully implemented comet on Python in a production system? How did you go about doing it and where can I find resources to implement my own?
Answers:
I haven’t done it, but this guy has and writes a good article about it, with Django examples and pointers (which I haven’t checked) to other frameworks.
I’ve done tons of APIs using twisted for stuff like that, most of which are available on my github account.
Most are client-side, but slosh is a server I wrote to do a realtime cheap pubsub sort of thing. It scales somewhat horizontally for reads by allowing for simple stream replication. Writes are a little different when you stick to plain HTTP, but I’ve pushed a decent amount through it for a demo.
Otherwise, you have full-on BOSH which most XMPP servers support and will allow you to decouple the message distribution from the web frontend.
Orbited seems as a nice solution. Haven’t tried it though.
Update: things have changed in the last 2.5 years.
We now have websockets in all major browsers, except IE (naturally) and a couple of very good abstractions over it, that provide many methods of emulating real-time communication.
- socket.io along with tornadio (socket.io 0.6) and tornadio2 (socket.io 0.7+)
- sock.js along with SockJS-tornado
I recommend you should use StreamHub Comet Server – its used by a lot of people – personally I use it with a couple of Django sites I run. You will need to write a tiny bit of Java to handle the streaming – I did this using Jython. The front-end code is some real simple Javascript a la:
StreamHub hub = new StreamHub();
hub.connect("http://myserver.com/");
hub.subscribe("newsfeed", function(sTopic, oData) { alert("new news item: " + oData.Title); });
The documentation is pretty good – I had similar problems as you trying to get started with the sparse docs of Cometd et al. For a start I’d read Getting Started With Comet and StreamHub, download and see how some of the examples work and reference the API docs if you need to:
Here is a full-featured example of combining Django, Orbited,and Twisted to create a real-time (Comet) app: http://github.com/clemesha/hotdot using Python.
the orbited and redis solutions are nice, but not longer relevant when you have something like the PubSubHubbub that google released. This makes it very easy to be the publisher or the subscriber to a given feed. http://code.google.com/p/pubsubhubbub/
Here’s an example that does long-polling with gevent and Django.
It uses greenlet – stack switching functionality from Stackless packaged as a CPython extension.
I am building a web application that has a real-time feed (similar to Facebook’s newsfeed) that I want to update via a long-polling mechanism. I understand that with Python, my choices are pretty much to either use Stackless (building from their Comet wsgi example) or Cometd + Twisted. Unfortunately there is very little documentation regarding these options and I cannot find good information online about production scale users of comet on Python.
Has anyone successfully implemented comet on Python in a production system? How did you go about doing it and where can I find resources to implement my own?
I haven’t done it, but this guy has and writes a good article about it, with Django examples and pointers (which I haven’t checked) to other frameworks.
I’ve done tons of APIs using twisted for stuff like that, most of which are available on my github account.
Most are client-side, but slosh is a server I wrote to do a realtime cheap pubsub sort of thing. It scales somewhat horizontally for reads by allowing for simple stream replication. Writes are a little different when you stick to plain HTTP, but I’ve pushed a decent amount through it for a demo.
Otherwise, you have full-on BOSH which most XMPP servers support and will allow you to decouple the message distribution from the web frontend.
Orbited seems as a nice solution. Haven’t tried it though.
Update: things have changed in the last 2.5 years.
We now have websockets in all major browsers, except IE (naturally) and a couple of very good abstractions over it, that provide many methods of emulating real-time communication.
- socket.io along with tornadio (socket.io 0.6) and tornadio2 (socket.io 0.7+)
- sock.js along with SockJS-tornado
I recommend you should use StreamHub Comet Server – its used by a lot of people – personally I use it with a couple of Django sites I run. You will need to write a tiny bit of Java to handle the streaming – I did this using Jython. The front-end code is some real simple Javascript a la:
StreamHub hub = new StreamHub();
hub.connect("http://myserver.com/");
hub.subscribe("newsfeed", function(sTopic, oData) { alert("new news item: " + oData.Title); });
The documentation is pretty good – I had similar problems as you trying to get started with the sparse docs of Cometd et al. For a start I’d read Getting Started With Comet and StreamHub, download and see how some of the examples work and reference the API docs if you need to:
Here is a full-featured example of combining Django, Orbited,and Twisted to create a real-time (Comet) app: http://github.com/clemesha/hotdot using Python.
the orbited and redis solutions are nice, but not longer relevant when you have something like the PubSubHubbub that google released. This makes it very easy to be the publisher or the subscriber to a given feed. http://code.google.com/p/pubsubhubbub/
Here’s an example that does long-polling with gevent and Django.
It uses greenlet – stack switching functionality from Stackless packaged as a CPython extension.