Kill or terminate subprocess when timeout?

I guess this is a common synchronization problem in event-oriented programming with threads and processes.

If you should always have only one subprocess running, make sure the current subprocess is killed before running the next one. Otherwise the signal handler may get a reference to the last subprocess run and ignore the older.

Suppose subprocess A is running. Before the alarm signal is handled, subprocess B is launched. Just after that, your alarm signal handler attempts to kill a subprocess. As the current PID (or the current subprocess pipe object) was set to B’s when launching the subprocess, B gets killed and A keeps running.

Is my guess correct?

To make your code easier to understand, I would include the part that creates a new subprocess just after the part that kills the current subprocess. That would make clear there is only one subprocess running at any time. The signal handler could do both the subprocess killing and launching, as if it was the iteration block that runs in a loop, in this case event-driven with the alarm signal every 1 second.

Here is something I wrote as a watchdog for subprocess execution. I use it now a lot, but I’m not so experienced so maybe there are some flaws in it:

import subprocess
import time

def subprocess_execute(command, time_out=60):
    """executing the command with a watchdog"""

    # launching the command
    c = subprocess.Popen(command)

    # now waiting for the command to complete
    t = 0
    while t < time_out and c.poll() is None:
        time.sleep(1)  # (comment 1)
        t += 1

    # there are two possibilities for the while to have stopped:
    if c.poll() is None:
        # in the case the process did not complete, we kill it
        c.terminate()
        # and fill the return code with some error value
        returncode = -1  # (comment 2)

    else:                 
        # in the case the process completed normally
        returncode = c.poll()

    return returncode   

Usage:

 return = subprocess_execute(['java', '-jar', 'some.jar'])

Comments:

1. here, the watchdog time out is in seconds; but it’s easy to change to whatever needed by changing the time.sleep() value. The time_out will have to be documented accordingly;
2. according to what is needed, here it maybe more suitable to raise some exception.

Documentation: I struggled a bit with the documentation of subprocess module to understand that subprocess.Popen is not blocking; the process is executed in parallel (maybe I do not use the correct word here, but I think it’s understandable).

But as what I wrote is linear in its execution, I really have to wait for the command to complete, with a time out to avoid bugs in the command to pause the nightly execution of the script.

Here’s what I use:

class KillerThread(threading.Thread):
  def __init__(self, pid, timeout, event ):
    threading.Thread.__init__(self)
    self.pid = pid
    self.timeout = timeout
    self.event = event
    self.setDaemon(True)
  def run(self):
    self.event.wait(self.timeout)
    if not self.event.isSet() :
      try:
        os.kill( self.pid, signal.SIGKILL )
      except OSError, e:
        #This is raised if the process has already completed
        pass    

def runTimed(dt, dir, args, kwargs ):
  event = threading.Event()
  cwd = os.getcwd()
  os.chdir(dir)
  proc = subprocess.Popen(args, **kwargs )
  os.chdir(cwd)
  killer = KillerThread(proc.pid, dt, event)
  killer.start()

  (stdout, stderr) = proc.communicate()
  event.set()      

  return (stdout,stderr, proc.returncode)

A bit more complex, I added an answer to solve a similar problem: Capturing stdout, feeding stdin, and being able to terminate after some time of inactivity and/or after some overall runtime.

subprocess.run was added in Python 3.5, meaning most systems should already have access to an up-to-date Python version with this method. It makes it trivial to set the timeout.

Here’s an example to run your-executable (assuming it’s in the working directory) with --optional args arguments and a timeout of 60 seconds:

import subprocess

subprocess.run(['./your-executable', '--optional', 'args'], timeout=60)

run will raise subprocess.TimeoutExpired if the timeout expires, which you could also handle:

try:
    subprocess.run(['./your-executable', '--optional', 'args'], timeout=60)
except subprocess.TimeoutExpired:
    print('timed out!')