Reraise exception from subprocess

Question

I have code executing in a subprocess that is supposed to raise an exception.

I would like to raise the same exception in the main process when the exception is returned from the subprocess (preferably while preserving the stack trace) but I am not sure how to do this.

I capture the stderr from the subprocess just fine but I can’t find how to parse it so I get the type of exception. How would I accomplish this?

I am using python 2.7

main method

import subprocess

example=subprocess.Popen(["python","example.py"],
                                    stdout = subprocess.PIPE,
                                    stderr = subprocess.PIPE)
return_obj, return_error = example.communicate()

if return_error:
# replace with error from subprocess
  print "Should raise ",NameError('HiThere')
  raise TypeError('Wrong type')

subprocess

raise NameError('HiThere')

Asked By: Bomaz

||

Source

Answer 1

The subprocess executes a different native application. It could be java, C++, lisp or even Fortran or Cobol. So you have only 2 ways to get the exception from a Python subprocess :

forget the subprocess at all, and directly call python code in the same python program, at simple try except will do the work
define in interface contract between the main and subprocess, for example requiring that the error output must be empty if no fatal exeption occurs and else contains a pickle of the exception and the stacktrace .

Answered By: Serge Ballesta

Answer 2

If all you want is your python code running in a separate process, you should probably not use subprocess. As Serge Ballesta said, subprocess‘ purpose is to run a different program at the OS level, without particularly caring for what it is – there’s nothing to help you properly handle the intricacies of a python interpreter process.

For this kind of purpose, it’s probably best to simply import your code and use multiprocessing, which exposes a high-level interface to help you run python code in multiple processes.

Assuming you have a well defined main function on example.py:

from examply import main as example_main
import multiprocessing
pool= multiprocessing.Pool(1)
pool.apply( example_main )

Using this code, both exceptions and return values will be transparently given to your main process.

You can also use Pool.apply_async, if you don’t want to block waiting for the result.

Answered By: loopbackbee

Answer 3

If both processes are python processes, and have to use pipes, exception object can be serialized through the pipe.

Caller:

parent_conn.send(event)
result = parent_conn.recv()
if isinstance(result, Exception):
    raise result
else:
    return result

Receiver:

def subprocess_loop(parent_conn, child_conn):
    
    parent_conn.close()

    while True:
        event = child_conn.recv()
        try:
            result = event_handler(event)
        except Exception as exc:
            child_conn.send(exc)
            continue
        child_conn.send(result)

Receiver process creation:

parent_conn, child_conn = Pipe()
process = Process(target=subprocess_loop, args=(parent_conn, child_conn))
process.start()
child_conn.close()

Answered By: Andrey

Reraise exception from subprocess

Question:

Answers: