Recommended Python publish/subscribe/dispatch module?
Question:
From PyPubSub:
Pypubsub provides a simple way for
your Python application to decouple
its components: parts of your
application can publish messages (with
or without data) and other parts can
subscribe/receive them. This allows
message “senders” and message
“listeners” to be unaware of each
other:
- one doesn’t need to import the other
- a sender doesn’t need to know
- “who” gets the messages,
- what the listeners will do with the data,
- or even if any listener will get the message data.
- similarly, listeners don’t need to worry about where messages come from.
This is a great tool for implementing
a Model-View-Controller architecture
or any similar architecture that
promotes decoupling of its components.
There seem to be quite a few Python modules for publishing/subscribing floating around the web, from PyPubSub, to PyDispatcher to simple “home-cooked” classes.
Are there specific advantages and disadvantages when comparing different different modules? Which sets of modules have been benchmarked and compared?
Thanks in advance
Answers:
PyDispatcher is used heavily in Django and it’s working perfectly for me (and for whole Django community, I guess).
As I remember, there are some performance issues:
- Arguments checking made by PyDispatcher is slow.
- Unused connections have unnecessary overhead.
AFAIK it’s very unlikely you will run into this issues in a small-to-medium sized application. So these issues may not concern you. If you think you need every pound of performance (premature optimization is the root of all evil!), you can look at modifications done to PyDispatcher in Django.
Hope this helps.
I recently looked carefully at py-amqplib to act as an AMQP client to a RabbitMQ broker. The latter tool is written in Erlang.
If you’re looking to decouple your app. then why couple it to the language itself? Consider using message queues which are language neutral and then you’ve really got room to grow!
That being said, AMQP takes effort to understand and may be more than you are willing to take on if your app. is working just fine as is. YMMV.
There is also the libraries by PJ Eby, RuleDispatch and the PEAK project, specially Trellis. I don’t know what their status actually but the mailing list is quite active.
Last version of Trellis on PyPi
I have also used the components from the Kamaelia project of the BBC. Axon is an interesting approach, but more component than publisher-consumer inspired. Well, its website is somewhat not up-to-date at all… There was a project or 2 in the Google SoC 2008 and work is being done.
Don’t know if it help 🙂
Edit : I just found Py-notify which is an “unorthodox” implementation of the Observer pattern. It has most of the functionalities that I need for my own tools.
The fact alone that PyPubSub seems to be a somewhat chaotically managed project (the Wiki on SF is dead, the website (another Wiki) which is linked on SF is currently broken) would be enough reason for me not to use it.
PyDispatcher has an intact website, but the only documentation they seem to provide is the one for the API generated from the docstrings. No traffic on the mailing list either… a bad sign!
As Mike also mentioned, it’s perfectly possible to choose a solution that is independent of Python. Now don’t get me wrong, I love Python, but still, in this field it can make sense use a framework that is decoupled from the programming language.
I’m not experienced with messaging, but I’m planning to have a look into a few solutions. So far these two (free, open source) projects seem to be the most promising for me (coincidentally, both are Apache projects):
Both seem to be reasonably matured projects, at least a far as documentation and community. I can’t comment on the software’s quality though, as I said, I didn’t use any of the software.
Qpid ships with client libraries for Python, but you could also use py-amqplib. For ActiveMQ there’s pyactivemq, which you can use to connect either via STOMP (Streaming Text Orientated Messaging Protocol) or via Openwire.
Some libraries I have found that haven’t yet been mentioned:
- Circuits – a Lightweight, Event driven Framework with a strong Component Architecture.
- C# Event Recipe
The best dispatch package for python seems to be the dispatch module inside django (called signals in the documentation). It is independent of the rest of django, and is short, documented, tested and very well written.
Edit: I forked this project into an independent signal project for Python.
Here is a newer one: https://github.com/shaunduncan/smokesignal. “smokesignal is a simple python library for sending and receiving signals. It draws some inspiration from the django signal framework but is meant as a general purpose variant.” Example:
from time import sleep
import smokesignal
@smokesignal.on('debug')
def verbose(val):
print "#", val
def main():
for i in range(100):
if i and i%10==0:
smokesignal.emit('debug', i)
sleep(.1)
main()
From PyPubSub:
Pypubsub provides a simple way for
your Python application to decouple
its components: parts of your
application can publish messages (with
or without data) and other parts can
subscribe/receive them. This allows
message “senders” and message
“listeners” to be unaware of each
other:
- one doesn’t need to import the other
- a sender doesn’t need to know
- “who” gets the messages,
- what the listeners will do with the data,
- or even if any listener will get the message data.
- similarly, listeners don’t need to worry about where messages come from.
This is a great tool for implementing
a Model-View-Controller architecture
or any similar architecture that
promotes decoupling of its components.
There seem to be quite a few Python modules for publishing/subscribing floating around the web, from PyPubSub, to PyDispatcher to simple “home-cooked” classes.
Are there specific advantages and disadvantages when comparing different different modules? Which sets of modules have been benchmarked and compared?
Thanks in advance
PyDispatcher is used heavily in Django and it’s working perfectly for me (and for whole Django community, I guess).
As I remember, there are some performance issues:
- Arguments checking made by PyDispatcher is slow.
- Unused connections have unnecessary overhead.
AFAIK it’s very unlikely you will run into this issues in a small-to-medium sized application. So these issues may not concern you. If you think you need every pound of performance (premature optimization is the root of all evil!), you can look at modifications done to PyDispatcher in Django.
Hope this helps.
I recently looked carefully at py-amqplib to act as an AMQP client to a RabbitMQ broker. The latter tool is written in Erlang.
If you’re looking to decouple your app. then why couple it to the language itself? Consider using message queues which are language neutral and then you’ve really got room to grow!
That being said, AMQP takes effort to understand and may be more than you are willing to take on if your app. is working just fine as is. YMMV.
There is also the libraries by PJ Eby, RuleDispatch and the PEAK project, specially Trellis. I don’t know what their status actually but the mailing list is quite active.
Last version of Trellis on PyPi
I have also used the components from the Kamaelia project of the BBC. Axon is an interesting approach, but more component than publisher-consumer inspired. Well, its website is somewhat not up-to-date at all… There was a project or 2 in the Google SoC 2008 and work is being done.
Don’t know if it help 🙂
Edit : I just found Py-notify which is an “unorthodox” implementation of the Observer pattern. It has most of the functionalities that I need for my own tools.
The fact alone that PyPubSub seems to be a somewhat chaotically managed project (the Wiki on SF is dead, the website (another Wiki) which is linked on SF is currently broken) would be enough reason for me not to use it.
PyDispatcher has an intact website, but the only documentation they seem to provide is the one for the API generated from the docstrings. No traffic on the mailing list either… a bad sign!
As Mike also mentioned, it’s perfectly possible to choose a solution that is independent of Python. Now don’t get me wrong, I love Python, but still, in this field it can make sense use a framework that is decoupled from the programming language.
I’m not experienced with messaging, but I’m planning to have a look into a few solutions. So far these two (free, open source) projects seem to be the most promising for me (coincidentally, both are Apache projects):
Both seem to be reasonably matured projects, at least a far as documentation and community. I can’t comment on the software’s quality though, as I said, I didn’t use any of the software.
Qpid ships with client libraries for Python, but you could also use py-amqplib. For ActiveMQ there’s pyactivemq, which you can use to connect either via STOMP (Streaming Text Orientated Messaging Protocol) or via Openwire.
Some libraries I have found that haven’t yet been mentioned:
- Circuits – a Lightweight, Event driven Framework with a strong Component Architecture.
- C# Event Recipe
The best dispatch package for python seems to be the dispatch module inside django (called signals in the documentation). It is independent of the rest of django, and is short, documented, tested and very well written.
Edit: I forked this project into an independent signal project for Python.
Here is a newer one: https://github.com/shaunduncan/smokesignal. “smokesignal is a simple python library for sending and receiving signals. It draws some inspiration from the django signal framework but is meant as a general purpose variant.” Example:
from time import sleep
import smokesignal
@smokesignal.on('debug')
def verbose(val):
print "#", val
def main():
for i in range(100):
if i and i%10==0:
smokesignal.emit('debug', i)
sleep(.1)
main()