Using RabbitMQ is there a way to look at the queue contents without a dequeue operation?
Question:
As a way to learn RabbitMQ and python I’m working on a project that allows me to distribute h264 encodes between a number of computers. The basics are done, I have a daemon that runs on Linux or Mac that attaches to queue, accepts jobs and encodes them using HandBrakeCLI and acks the message once the encode is complete. I’ve also built a simple tool to push items into the queue.
Now I want to expand the capabilities of the tool that pushes items into the queue so that I can view what is in the queue. I’m aware of the ability to see how many items are in the queue, but I want to be able to get the actual messages so I can show what movie or TV show is waiting to be encoded yet. The idea is that the queue manager would receive messages from the encoder clients when a job has completed and then refresh the queue list.
I know there is a convoluted way of keeping the queue manager’s list in sync with the actual work queue but I’d like this to be “persistent” in that I should be able to close the queue manager and reopen it later to see the queue.
Answers:
What you want to do is called browsing the queue, although I gather from this that RabbitMQ does not yet support that.
Queue browsing is not supported directly, but if you declare a queue with NO auto acknowledgements and do not ACK the messages that you receive, then you can see everything in it. After you have had a look, send a CANCEL on the channel, or disconnect and reconnect to cause all the messages to be requeued. This does increment a number in the message headers, but otherwise leaves the messages untouched.
I built an app where message ordering was not terribly important, and I frequently scanned through the queue in this way. If I found a problem, I would dump the messages into a file, fix them and resubmit.
If you only need to peek at a message or two once in a while you can do that with the RabbitMQ management plugin.
In addition, if you only need a message count, you can get that every time you declare the queue, or on a basic.get command.
@MichaelDillon based on your answer to make others life easier I am putting here a no_ack example:
#!/usr/bin/env python
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='Q.hello')
def callback(ch, method, properties, body):
print(" [x] Received %r" % body)
# ch.basic_ack(delivery_tag=method.delivery_tag)
channel.basic_consume(callback, queue='Q.hello')
print(' [*] Waiting for messages. To exit press CTRL+C')
channel.start_consuming()
As a way to learn RabbitMQ and python I’m working on a project that allows me to distribute h264 encodes between a number of computers. The basics are done, I have a daemon that runs on Linux or Mac that attaches to queue, accepts jobs and encodes them using HandBrakeCLI and acks the message once the encode is complete. I’ve also built a simple tool to push items into the queue.
Now I want to expand the capabilities of the tool that pushes items into the queue so that I can view what is in the queue. I’m aware of the ability to see how many items are in the queue, but I want to be able to get the actual messages so I can show what movie or TV show is waiting to be encoded yet. The idea is that the queue manager would receive messages from the encoder clients when a job has completed and then refresh the queue list.
I know there is a convoluted way of keeping the queue manager’s list in sync with the actual work queue but I’d like this to be “persistent” in that I should be able to close the queue manager and reopen it later to see the queue.
What you want to do is called browsing the queue, although I gather from this that RabbitMQ does not yet support that.
Queue browsing is not supported directly, but if you declare a queue with NO auto acknowledgements and do not ACK the messages that you receive, then you can see everything in it. After you have had a look, send a CANCEL on the channel, or disconnect and reconnect to cause all the messages to be requeued. This does increment a number in the message headers, but otherwise leaves the messages untouched.
I built an app where message ordering was not terribly important, and I frequently scanned through the queue in this way. If I found a problem, I would dump the messages into a file, fix them and resubmit.
If you only need to peek at a message or two once in a while you can do that with the RabbitMQ management plugin.
In addition, if you only need a message count, you can get that every time you declare the queue, or on a basic.get command.
@MichaelDillon based on your answer to make others life easier I am putting here a no_ack example:
#!/usr/bin/env python
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='Q.hello')
def callback(ch, method, properties, body):
print(" [x] Received %r" % body)
# ch.basic_ack(delivery_tag=method.delivery_tag)
channel.basic_consume(callback, queue='Q.hello')
print(' [*] Waiting for messages. To exit press CTRL+C')
channel.start_consuming()