An enterprise scheduler for python (like quartz)
Question:
I am looking for an enterprise tasks scheduler for python, like quartz is for Java.
Requirements:
- Persistent: if the process restarts or the machine restarts, then all the jobs must stay there and must be fired after restarting.
- Jobs must enter and exit the scheduler in a transaction (i.e. if some database operation fail, in a database unrelated to the scheduler, then the job must not have exited or entered the scheduler).
- Scalability. Depends on the measure of success of the project, but I would prefer to know from the beggining that I am not starting from a dead end.
- Configurability: when tasks expire, how many can be fired simultaneously, etc.
Any recommendations? Is there something python specific, or is it possible (or even good) to interface to Quartz from python.
Answers:
Is APScheduler what you’re looking for?
We‘re using Sun Grid Engine, which is accessible through DRMAA, which happens to have Python bindings.
You can use Celery
Celery is an asynchronous task queue/job queue based on distributed
message passing. It is focused on real-time operation, but supports
scheduling as well.
Install Celery using pip install celery
Another option is using RQ.
RQ (Redis Queue) is a simple Python library for queueing jobs and
processing them in the background with workers. It is backed by Redis
and it is designed to have a low barrier to entry. It should be
integrated in your web stack easily.
Install using pip install rq.
This project seems not bad. a flexible python library for building your own cron-like system to schedule jobs, which is to run a tornado process to serve REST APIs and a web ui.
I am looking for an enterprise tasks scheduler for python, like quartz is for Java.
Requirements:
- Persistent: if the process restarts or the machine restarts, then all the jobs must stay there and must be fired after restarting.
- Jobs must enter and exit the scheduler in a transaction (i.e. if some database operation fail, in a database unrelated to the scheduler, then the job must not have exited or entered the scheduler).
- Scalability. Depends on the measure of success of the project, but I would prefer to know from the beggining that I am not starting from a dead end.
- Configurability: when tasks expire, how many can be fired simultaneously, etc.
Any recommendations? Is there something python specific, or is it possible (or even good) to interface to Quartz from python.
Is APScheduler what you’re looking for?
We‘re using Sun Grid Engine, which is accessible through DRMAA, which happens to have Python bindings.
You can use Celery
Celery is an asynchronous task queue/job queue based on distributed
message passing. It is focused on real-time operation, but supports
scheduling as well.
Install Celery using pip install celery
Another option is using RQ.
RQ (Redis Queue) is a simple Python library for queueing jobs and
processing them in the background with workers. It is backed by Redis
and it is designed to have a low barrier to entry. It should be
integrated in your web stack easily.
Install using pip install rq.
This project seems not bad. a flexible python library for building your own cron-like system to schedule jobs, which is to run a tornado process to serve REST APIs and a web ui.