How can I use Celery in Django with just the DB?
Question:
Looking at https://docs.celeryq.dev/en/v5.2.7/getting-started/backends-and-brokers/index.html it sounds pretty much as if it’s not possible / desirable. There is a section about SQLAlchemy, but Django does not use SQLAlchemy.
In way older docs, there is https://docs.celeryq.dev/en/3.1/getting-started/brokers/django.html .
Is it possible with recent Celery / Django versions to use Celery with just the database for storing messages / results?
Answers:
If you really want to use a relational SQL database as a broker, SQLAlchemy indeed is your best option, since it’s supported by Celery already. I would be careful about this approach though, because not only you’re (ab)using a relational database as a message broker, but also the Celery docs recommend caution:
Historically, SQLAlchemy has not been the most stable result backend
so if chosen one should proceed with caution.
Django does not use SQLAlchemy indeed, because it has it’s own ORM, but that’s not preventing you from using SQLAlchemy in the Celery worker.
Yes you can totally do this, even if it’s not the most performant/recommended way to do. I use it for simple projects in which I don’t want to add Redis.
To do so, first, add SQLAlchemy v1 as a dependency in your project: SQLAlchemy = "1.*"
Then in your settings.py:
- if you use PostgreSQL:
CELERY_BROKER_URL = sqla+postgresql://user:[email protected]:5432/dbname
- if you use SQLite:
CELERY_BROKER_URL = "sqla+sqlite:///" + os.path.join(BASE_DIR, 'your_database.db') . Note that the folder holding the database must be writable. For example if your database is located in project/dbfolder/database.db, chmod 777 project/dbfolder will do the trick.
As a sidenote, I’m using django-celery-results to store results of my tasks. This way, I have a fully featured Celery without using other tech tool (like rabbitMQ or Redis) in my stack.
Looking at https://docs.celeryq.dev/en/v5.2.7/getting-started/backends-and-brokers/index.html it sounds pretty much as if it’s not possible / desirable. There is a section about SQLAlchemy, but Django does not use SQLAlchemy.
In way older docs, there is https://docs.celeryq.dev/en/3.1/getting-started/brokers/django.html .
Is it possible with recent Celery / Django versions to use Celery with just the database for storing messages / results?
If you really want to use a relational SQL database as a broker, SQLAlchemy indeed is your best option, since it’s supported by Celery already. I would be careful about this approach though, because not only you’re (ab)using a relational database as a message broker, but also the Celery docs recommend caution:
Historically, SQLAlchemy has not been the most stable result backend
so if chosen one should proceed with caution.
Django does not use SQLAlchemy indeed, because it has it’s own ORM, but that’s not preventing you from using SQLAlchemy in the Celery worker.
Yes you can totally do this, even if it’s not the most performant/recommended way to do. I use it for simple projects in which I don’t want to add Redis.
To do so, first, add SQLAlchemy v1 as a dependency in your project: SQLAlchemy = "1.*"
Then in your settings.py:
- if you use PostgreSQL:
CELERY_BROKER_URL = sqla+postgresql://user:[email protected]:5432/dbname - if you use SQLite:
CELERY_BROKER_URL = "sqla+sqlite:///" + os.path.join(BASE_DIR, 'your_database.db'). Note that the folder holding the database must be writable. For example if your database is located inproject/dbfolder/database.db,chmod 777 project/dbfolderwill do the trick.
As a sidenote, I’m using django-celery-results to store results of my tasks. This way, I have a fully featured Celery without using other tech tool (like rabbitMQ or Redis) in my stack.