Celery: When should you choose Redis as a message broker over RabbitMQ?
Question:
My rough understanding is that Redis is better if you need the in-memory key-value store feature, however I am not sure how that has anything to do with distributing tasks?
Does that mean we should use Redis as a message broker IF we are already using it for something else?
Answers:
I’ve used both recently (2017-2018), and they are both super stable with Celery 4. So your choice can be based on the details of your hosting setup.
- If you must use Celery version 2 or version 3, go with RabbitMQ. Otherwise…
- If you are using Redis for any other reason, go with Redis
- If you are hosting at AWS, go with Redis so that you can use a managed Redis as service
- If you hate complicated installs, go with Redis
- If you already have RabbitMQ installed, stay with RabbitMQ
In the past, I would have recommended RabbitMQ because it was more stable and easier to setup with Celery than Redis, but I don’t believe that’s true any more.
Update 2019
AWS now has a managed service that is equivalent to RabbitMQ called Amazon MQ, which could reduce the headache of running this as a service in production. Please comment below if you have any experience with this and celery.
The Redis broker gives tasks to workers in a fair round robin between different queues. Rabbit is FIFO always. For me, a fair round robin was preferable and I tried both. Rabbit seems a tad more stable though.
My rough understanding is that Redis is better if you need the in-memory key-value store feature, however I am not sure how that has anything to do with distributing tasks?
Does that mean we should use Redis as a message broker IF we are already using it for something else?
I’ve used both recently (2017-2018), and they are both super stable with Celery 4. So your choice can be based on the details of your hosting setup.
- If you must use Celery version 2 or version 3, go with RabbitMQ. Otherwise…
- If you are using Redis for any other reason, go with Redis
- If you are hosting at AWS, go with Redis so that you can use a managed Redis as service
- If you hate complicated installs, go with Redis
- If you already have RabbitMQ installed, stay with RabbitMQ
In the past, I would have recommended RabbitMQ because it was more stable and easier to setup with Celery than Redis, but I don’t believe that’s true any more.
Update 2019
AWS now has a managed service that is equivalent to RabbitMQ called Amazon MQ, which could reduce the headache of running this as a service in production. Please comment below if you have any experience with this and celery.
The Redis broker gives tasks to workers in a fair round robin between different queues. Rabbit is FIFO always. For me, a fair round robin was preferable and I tried both. Rabbit seems a tad more stable though.