Run a program from python, and have it continue to run after the script is killed
Question:
I’ve tried running things like this:
subprocess.Popen(['nohup', 'my_command'],
stdout=open('/dev/null', 'w'),
stderr=open('logfile.log', 'a'))
This works if the parent script exits gracefully, but if I kill the script (Ctrl-C), all my child processes are killed too. Is there a way to avoid this?
The platforms I care about are OS X and Linux, using Python 2.6 and Python 2.7.
Answers:
The usual way to do this on Unix systems is to fork and exit if you’re the parent. Have a look at os.fork()
.
Here’s a function that does the job:
def spawnDaemon(func):
# do the UNIX double-fork magic, see Stevens' "Advanced
# Programming in the UNIX Environment" for details (ISBN 0201563177)
try:
pid = os.fork()
if pid > 0:
# parent process, return and keep running
return
except OSError, e:
print >>sys.stderr, "fork #1 failed: %d (%s)" % (e.errno, e.strerror)
sys.exit(1)
os.setsid()
# do second fork
try:
pid = os.fork()
if pid > 0:
# exit from second parent
sys.exit(0)
except OSError, e:
print >>sys.stderr, "fork #2 failed: %d (%s)" % (e.errno, e.strerror)
sys.exit(1)
# do stuff
func()
# all done
os._exit(os.EX_OK)
The child process receives the same SIGINT
as your parent process because it’s in the same process group. You can put the child in its own process group by calling os.setpgrp()
in the child process. Popen
‘s preexec_fn
argument is useful here:
subprocess.Popen(['nohup', 'my_command'],
stdout=open('/dev/null', 'w'),
stderr=open('logfile.log', 'a'),
preexec_fn=os.setpgrp
)
(preexec_fn
is for un*x-oids only. There appears to be a rough equivalent for Windows “creationflags=CREATE_NEW_PROCESS_GROUP
“, but I’ve never tried it.)
with open('/dev/null', 'w') as stdout, open('logfile.log', 'a') as stderr:
subprocess.Popen(['my', 'command'], stdout=stdout, stderr=stderr)
class subprocess.Popen(…)
Execute a child program in a new process.
On POSIX, the class uses os.execvp()-like behavior to execute the
child program. On Windows, the class uses the Windows CreateProcess()
function.
os.execvpe(file, args, env)
These functions all execute a new program, replacing the current
process; they do not return. On Unix, the new executable is loaded
into the current process, and will have the same process id as the
caller. Errors will be reported as OSError exceptions.
After an hour of various attempts, this works for me:
process = subprocess.Popen(["someprocess"], creationflags=subprocess.DETACHED_PROCESS | subprocess.CREATE_NEW_PROCESS_GROUP)
It’s solution for windows.
Since 3.2 you can also use start_new_session
flag (POSIX only).
import subprocess
p = subprocess.Popen(["sleep", "60"], start_new_session=True)
ret = p.wait()
I’ve tried running things like this:
subprocess.Popen(['nohup', 'my_command'],
stdout=open('/dev/null', 'w'),
stderr=open('logfile.log', 'a'))
This works if the parent script exits gracefully, but if I kill the script (Ctrl-C), all my child processes are killed too. Is there a way to avoid this?
The platforms I care about are OS X and Linux, using Python 2.6 and Python 2.7.
The usual way to do this on Unix systems is to fork and exit if you’re the parent. Have a look at os.fork()
.
Here’s a function that does the job:
def spawnDaemon(func):
# do the UNIX double-fork magic, see Stevens' "Advanced
# Programming in the UNIX Environment" for details (ISBN 0201563177)
try:
pid = os.fork()
if pid > 0:
# parent process, return and keep running
return
except OSError, e:
print >>sys.stderr, "fork #1 failed: %d (%s)" % (e.errno, e.strerror)
sys.exit(1)
os.setsid()
# do second fork
try:
pid = os.fork()
if pid > 0:
# exit from second parent
sys.exit(0)
except OSError, e:
print >>sys.stderr, "fork #2 failed: %d (%s)" % (e.errno, e.strerror)
sys.exit(1)
# do stuff
func()
# all done
os._exit(os.EX_OK)
The child process receives the same SIGINT
as your parent process because it’s in the same process group. You can put the child in its own process group by calling os.setpgrp()
in the child process. Popen
‘s preexec_fn
argument is useful here:
subprocess.Popen(['nohup', 'my_command'],
stdout=open('/dev/null', 'w'),
stderr=open('logfile.log', 'a'),
preexec_fn=os.setpgrp
)
(preexec_fn
is for un*x-oids only. There appears to be a rough equivalent for Windows “creationflags=CREATE_NEW_PROCESS_GROUP
“, but I’ve never tried it.)
with open('/dev/null', 'w') as stdout, open('logfile.log', 'a') as stderr:
subprocess.Popen(['my', 'command'], stdout=stdout, stderr=stderr)
class subprocess.Popen(…)
Execute a child program in a new process.
On POSIX, the class uses os.execvp()-like behavior to execute the
child program. On Windows, the class uses the Windows CreateProcess()
function.os.execvpe(file, args, env)
These functions all execute a new program, replacing the current
process; they do not return. On Unix, the new executable is loaded
into the current process, and will have the same process id as the
caller. Errors will be reported as OSError exceptions.
After an hour of various attempts, this works for me:
process = subprocess.Popen(["someprocess"], creationflags=subprocess.DETACHED_PROCESS | subprocess.CREATE_NEW_PROCESS_GROUP)
It’s solution for windows.
Since 3.2 you can also use start_new_session
flag (POSIX only).
import subprocess
p = subprocess.Popen(["sleep", "60"], start_new_session=True)
ret = p.wait()