Capture real time `stdout` and `stderr` when run a function in a process python

Question:

I have a python function and want to run it as a separate process with multiprocessing package.

def run(ctx: Context):
    print("hello world!")
    return ctx

afterward running it as a separate process with the following script:

import multiprocessing

p = multiprocessing.Process(target=run, args=(ctx, ))
p.start()
p.join()

Now, I need to capture live stdout and stderr of the above process. Is there any way like as:

import subprocess
proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE)
while True:
  line = proc.stdout.readline()
  if not line:
    break

But I need to pass the function not running a command with Popen. Do you know how can I read stdout when I run my function in a separate process?

Asked By: Maryam

||

Answers:

You can capture the live stdout and stderr of the multiprocessing process using the multiprocessing module.

Here’s how you can do it:

import multiprocessing
import sys

def run(ctx: Context, pipe):
    old_stdout = sys.stdout
    sys.stdout = pipe
    print("hello world!")
    sys.stdout = old_stdout
    return ctx

parent_conn, child_conn = multiprocessing.Pipe()

p = multiprocessing.Process(target=run, args=(ctx, child_conn))
p.start()

while True:
    line = parent_conn.recv()
    if line == "EOF":
        break
    print(line)

p.join()

You can then use the parent_conn pipe to receive data sent from the child process, in this case run(). You can also use the child_conn pipe to send data to the child process.

In this example, the standard output of the child process is redirected to the child_conn pipe, and then you can receive it in the main process using the parent_conn pipe.

Answered By: Amin