How do I run another script in Python without waiting for it to finish?
Question:
I am creating a little dashboard for a user that will allow him to run specific jobs. I am using Django so I want him to be able to click a link to start the job and then return the page back to him with a message that the job is running. The results of the job will be emailed to him later.
I believe I am supposed to use subprocess.Popen but I’m not sure of that. So in pseudocode, here is what I want to do:
if job == 1:
run script in background: /path/to/script.py
return 'Job is running'
Answers:
p = subprocess.Popen([sys.executable, '/path/to/script.py'],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
That will start the subprocess in background. Your script will keep running normally.
Read the documentation here.
subprocess.Popen is indeed what you are looking for.
Although if you find that you want to start communicating a bunch of information between the subprocess and the parent, you may want to consider a thread, or RPC framework like Twisted.
But most likely those are too heavy for your application.
Running this through a message queue is definitely the way to go if you’re thinking about long-term scaling. Send a message to the queue who’s running constantly in the background, and write job handlers to deal with the different sorts of messages.
Since you’re using Django, I think Beanstalkd is a pretty good fit. Here’s a pretty nice tutorial on the subject. The first comment in that article also has some good tips.
Personally I’ve rolled with a custom in-memory queue server written in Erlang, with Python-bindings written in C. But redis looks like it might work out as a great contender for future queuing/messaging-needs. Hope this helps!
I am creating a little dashboard for a user that will allow him to run specific jobs. I am using Django so I want him to be able to click a link to start the job and then return the page back to him with a message that the job is running. The results of the job will be emailed to him later.
I believe I am supposed to use subprocess.Popen but I’m not sure of that. So in pseudocode, here is what I want to do:
if job == 1:
run script in background: /path/to/script.py
return 'Job is running'
p = subprocess.Popen([sys.executable, '/path/to/script.py'],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
That will start the subprocess in background. Your script will keep running normally.
Read the documentation here.
subprocess.Popen is indeed what you are looking for.
Although if you find that you want to start communicating a bunch of information between the subprocess and the parent, you may want to consider a thread, or RPC framework like Twisted.
But most likely those are too heavy for your application.
Running this through a message queue is definitely the way to go if you’re thinking about long-term scaling. Send a message to the queue who’s running constantly in the background, and write job handlers to deal with the different sorts of messages.
Since you’re using Django, I think Beanstalkd is a pretty good fit. Here’s a pretty nice tutorial on the subject. The first comment in that article also has some good tips.
Personally I’ve rolled with a custom in-memory queue server written in Erlang, with Python-bindings written in C. But redis looks like it might work out as a great contender for future queuing/messaging-needs. Hope this helps!