"subprocess.Popen" – checking for success and errors

Question:

I want to check if a subprocess has finished execution successfully or failed. Currently I have come up with a solution but I am not sure if it is correct and reliable. Is it guaranteed that every process outputs its errors only to stderr respectfully to stdout:

Note: I am not interested in just redirecting/printing out the output. That I know already how to do.

pipe = subprocess.Popen(command,
                                stdout=subprocess.PIPE,
                                stderr=subprocess.PIPE,
                                universal_newlines=True)

if "" == pipe.stdout.readline():
    print("Success")
    self.isCommandExectutionSuccessful = True

if not "" == pipe.stderr.readline():
    print("Error")
    self.isCommandExectutionSuccessful = True

alternatively:

   if "" == pipe.stdout.readline():
       print("Success")
       self.isCommandExectutionSuccessful = True
   else:
       print("Error")
       self.isCommandExectutionSuccessful = False

and:

   if not "" == pipe.stderr.readline():
       print("Success")
       self.isCommandExectutionSuccessful = True
   else:
       print("Error")
       self.isCommandExectutionSuccessful = False
Asked By: Zingam

||
Source

Answers:

      output,error=pipe.communicate()

This will wait for command to finish and give you output or error depending on the state of command.

Answered By: vks

Do you need to do anything with the output of the process?

The check_call method might be useful here. See the python docs here: https://docs.python.org/2/library/subprocess.html#subprocess.check_call

You can then use this as follows:

try:
  subprocess.check_call(command)
except subprocess.CalledProcessError:
  # There was an error - command exited with non-zero code

However, this relies on command returning an exit code of 0 for succesful completion and a non-zero value for an error.

If you need to capture the output as well, then the check_output method may be more appropriate. It is still possible to redirect the standard error if you need this as well.

try:
  proc = subprocess.check_output(command, stderr=subprocess.STDOUT)
  # do something with output
except subprocess.CalledProcessError:
  # There was an error - command exited with non-zero code

See the docs here: https://docs.python.org/2/library/subprocess.html#subprocess.check_output

Answered By: elParaguayo

You can check return code of the process using check_call() method.
In case if process returned non-zero value CalledProcessError will be raised.

Answered By: Alexey Bureev

This is how I did it finally:

    # Call a system process
    try:
        # universal_newlines - makes manual decoding of subprocess.stdout unnecessary
        output = subprocess.check_output(command,
                                         stderr=subprocess.STDOUT,
                                         universal_newlines=True)

        # Print out command's standard output (elegant)
        self.textEdit_CommandLineOutput.insertPlainText(output)
        self.isCommandExecutionSuccessful = True

    except subprocess.CalledProcessError as error:
        self.isCommandExecutionSuccessful = False

        errorMessage = ">>> Error while executing:n"
                       + command
                       + "n>>> Returned with error:n"
                       + str(error.output)
        self.textEdit_CommandLineOutput.append(errorMessage)

        QMessageBox.critical(None,
                             "ERROR",
                             errorMessage)
        print("Error: " + errorMessage)

    except FileNotFoundError as error:
        errorMessage = error.strerror
        QMessageBox.critical(None,
                             "ERROR",
                             errorMessage)
        print("Error: ", errorMessage)

I hope it will be useful to someone else.

Answered By: Zingam

Complete solution with check on return code, stdout and stderr:

import subprocess as sp

# ok
pipe = sp.Popen( 'ls /bin', shell=True, stdout=sp.PIPE, stderr=sp.PIPE )
# res = tuple (stdout, stderr)
res = pipe.communicate()
print("retcode =", pipe.returncode)
print("res =", res)
print("stderr =", res[1])
for line in res[0].decode(encoding='utf-8').split('n'):
  print(line)

# with error
pipe = sp.Popen( 'ls /bing', shell=True, stdout=sp.PIPE, stderr=sp.PIPE )
res = pipe.communicate()
print("retcode =", pipe.returncode)
print("res =", res)
print("stderr =", res[1])

Prints:

retcode = 0
res = (b'bashnbunzip2nbusyboxnbzcatn...zmorenznewn', b'')
stderr = b''
bash
bunzip2
busybox
bzcat
...
zmore
znew

retcode = 2
res = (b'', b"ls: cannot access '/bing': No such file or directoryn")
stderr = b"ls: cannot access '/bing': No such file or directoryn"
Answered By: PJ_Finnegan

All previous answers are correct but here is another way to skin a cat

(this is test with Python 3.9)

cmd_aws="aws s3 cp s3://some_bucket/somefile+" "+"somefile_name"
            proc_prep = subprocess.Popen(cmd_aws, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            out, err = proc_prep.communicate()
            if err:
                print(out)
                print(err)
Answered By: grepit

There are situations where using check_call is not a possibility. For example when you need to communicate with the process (e.g., passing input with communicate).

In this case, a simple solution is to mimic check_call manually. We can look at the Python source code to see what check_call is doing here and you can see it’s simply checking the return code is 0 and if not it raises a CalledProcessError. It couldn’t be simpler.

You may notice that check_call is not including stdout or stderr in the CalledProcessError even though it can accept them. This is because the process may not have captured them unless it was given subprocess.PIPE for stdout and stderr Popen parameters.

video_url = "http://.../sample/BigBuckBunny.mp4"

p = subprocess.Popen(
    [
        "ffplay",
        "-i", "-"
    ],
    stdin=subprocess.PIPE
)

p.communicate(video_url.encode())
if p.returncode != 0:
    raise subprocess.CalledProcessError(p.returncode, p.args)

The above is an example scenario where we need to PIPE input data (in this case an URL) to the process. It is not possible to write to stdin with check_call.

We simply mimic check_call with the last 2 lines.

Answered By: rlaphoenix