Why would running scheduled tasks with Celery be preferable over crontab?
Question:
Considering Celery is already a part of the stack to run task queues (i.e. it is not being added just for running crons, that seems an overkill IMHO ).
How can its “periodic tasks” feature be beneficial as a replacement for crontab ?
Specifically looking for following points.
- Major pros/cons over crontab
- Use cases where celery is better choice than crontab
- Django specific use case: Celery vs crontab to run django based periodic tasks, when celery has been included in the stack as
django-celery for queing django tasks.
Answers:
I’ve been using cron for a production website, and have switched to celery on a current project.
I’m far more into celery than cron, here is why:
- Celery + Celerybeat has finer granularity than cron. Cron cannot run more than once a minute, while celery can (I have a task run every 90 seconds which checks an email queue to send messages, and another which cleans the online users list).
- A cron line has to call a script or a unique command, with absolute path and user info. Celery calls python functions, no need to write more than code.
- With celery, to deploy to another machine, you generally just have to pull/copy your code, which is generally in one place. Deploying with cron would need more work (you can automate it but…)
- I really find celery better suited than cron for routine cleaning (cache, database), and in general, for short tasks. Dumping a database is more a work for cron, however, because you don’t want clutter the event queue with too long tasks.
- Not the least, Celery is easily distributed across machines.
Celery is indicated any time you need to coordinate jobs across multiple machines, ensure jobs run even as machines are added or dropped from a workgroup, have the ability to set expiration times for jobs, define multi-step jobs with graph-style rather than linear dependency flow, or have a single repository of scheduling logic that operates the same across multiple operating systems and versions.
Considering Celery is already a part of the stack to run task queues (i.e. it is not being added just for running crons, that seems an overkill IMHO ).
How can its “periodic tasks” feature be beneficial as a replacement for crontab ?
Specifically looking for following points.
- Major pros/cons over crontab
- Use cases where celery is better choice than crontab
- Django specific use case: Celery vs crontab to run django based periodic tasks, when celery has been included in the stack as
django-celeryfor queing django tasks.
I’ve been using cron for a production website, and have switched to celery on a current project.
I’m far more into celery than cron, here is why:
- Celery + Celerybeat has finer granularity than cron. Cron cannot run more than once a minute, while celery can (I have a task run every 90 seconds which checks an email queue to send messages, and another which cleans the online users list).
- A cron line has to call a script or a unique command, with absolute path and user info. Celery calls python functions, no need to write more than code.
- With celery, to deploy to another machine, you generally just have to pull/copy your code, which is generally in one place. Deploying with cron would need more work (you can automate it but…)
- I really find celery better suited than cron for routine cleaning (cache, database), and in general, for short tasks. Dumping a database is more a work for cron, however, because you don’t want clutter the event queue with too long tasks.
- Not the least, Celery is easily distributed across machines.
Celery is indicated any time you need to coordinate jobs across multiple machines, ensure jobs run even as machines are added or dropped from a workgroup, have the ability to set expiration times for jobs, define multi-step jobs with graph-style rather than linear dependency flow, or have a single repository of scheduling logic that operates the same across multiple operating systems and versions.