Capture real time `stdout` and `stderr` when run a function in a process python
Question:
I have a python function and want to run it as a separate process with multiprocessing
package.
def run(ctx: Context):
print("hello world!")
return ctx
afterward running it as a separate process with the following script:
import multiprocessing
p = multiprocessing.Process(target=run, args=(ctx, ))
p.start()
p.join()
Now, I need to capture live stdout
and stderr
of the above process. Is there any way like as:
import subprocess
proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE)
while True:
line = proc.stdout.readline()
if not line:
break
But I need to pass the function not running a command with Popen
. Do you know how can I read stdout
when I run my function in a separate process?
Answers:
You can capture the live stdout
and stderr
of the multiprocessing process using the multiprocessing
module.
Here’s how you can do it:
import multiprocessing
import sys
def run(ctx: Context, pipe):
old_stdout = sys.stdout
sys.stdout = pipe
print("hello world!")
sys.stdout = old_stdout
return ctx
parent_conn, child_conn = multiprocessing.Pipe()
p = multiprocessing.Process(target=run, args=(ctx, child_conn))
p.start()
while True:
line = parent_conn.recv()
if line == "EOF":
break
print(line)
p.join()
You can then use the parent_conn
pipe to receive data sent from the child process, in this case run(). You can also use the child_conn
pipe to send data to the child process.
In this example, the standard output of the child process is redirected to the child_conn
pipe, and then you can receive it in the main process using the parent_conn
pipe.
I have a python function and want to run it as a separate process with multiprocessing
package.
def run(ctx: Context):
print("hello world!")
return ctx
afterward running it as a separate process with the following script:
import multiprocessing
p = multiprocessing.Process(target=run, args=(ctx, ))
p.start()
p.join()
Now, I need to capture live stdout
and stderr
of the above process. Is there any way like as:
import subprocess
proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE)
while True:
line = proc.stdout.readline()
if not line:
break
But I need to pass the function not running a command with Popen
. Do you know how can I read stdout
when I run my function in a separate process?
You can capture the live stdout
and stderr
of the multiprocessing process using the multiprocessing
module.
Here’s how you can do it:
import multiprocessing
import sys
def run(ctx: Context, pipe):
old_stdout = sys.stdout
sys.stdout = pipe
print("hello world!")
sys.stdout = old_stdout
return ctx
parent_conn, child_conn = multiprocessing.Pipe()
p = multiprocessing.Process(target=run, args=(ctx, child_conn))
p.start()
while True:
line = parent_conn.recv()
if line == "EOF":
break
print(line)
p.join()
You can then use the parent_conn
pipe to receive data sent from the child process, in this case run(). You can also use the child_conn
pipe to send data to the child process.
In this example, the standard output of the child process is redirected to the child_conn
pipe, and then you can receive it in the main process using the parent_conn
pipe.