python subprocess does not write the output

Question

I have the following code snippet as a Python script. I get the proper job output files with no errors. However the stdout subprocess log files (i.e. COSMOSIS_output_XXX.txt etc) don’t store the expected runtime logs. Instead, these files have <_io.TextIOWrapper name=9 encoding='UTF-8'> as the only output written in the files. What is it I am doing wrong?

import subprocess
from subprocess import Popen
jobname = "cosmosis"
arg2 = "output.filename="
Vector = (
    " " + ini + " " + arg2 + COSMOSIS_PATH + os.path.splitext(ini)[0] + ".txt"
)                                                                                                                                                                                               

job3 = subprocess.Popen(
    ["cosmosis" + Vector],
    shell=True,
    text=True,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
)

file = open("%sCOSMOSIS_output_%s.txt" % (ERROR_PATH, ini), "a")
sys.stdout = file
print(job3.stdout)
file_e = open("%sCOSMOSIS_output_ERROR_%s.txt" % (ERROR_PATH, ini), "a")
sys.stdout = file_e
print(job3.stderr)
try:
    outs, errs = job3.communicate(timeout=2000)                                                                                                                                      
except TimeoutExpired:
    outs, errs = job3.communicate()
    job3.kill()
file.close()
file_e.close()

Asked By: Ayan Mitra

||

Source

Answer 1

First of all you are just opening the file, you are not reading anything from it, you are not storing the information of the file anywhere, so it will just create that <_io.TextIOWrapper name=9 encoding='UTF-8'> which is very easy reproducible:

file = open("testtextf.txt","a")
print(file)

You have to read it somehow for example with .read():

data = file.read()

Also i do not recommend using "open" unless you want to deal with files being left open if anything goes wrong in between the lines before you close it again.

I highly recommend to use "with open" instead.

Answered By: Aru

Answer 2

Ok, the following snippet is sufficient solution to the question posted above.

with open("%sCOSMOSIS_output_%s.txt" % (ERROR_PATH, ini), "wb") as file, open("%sCOSMOSIS_output_ERROR_%s.txt" % (ERROR_PATH, ini), "wb") as file_e:
    job3 = subprocess.Popen(
        ["cosmosis" + Vector],
        shell=True,
        text=True,
        stdout=file,
        stderr=file_e,
    )

Answered By: Ayan Mitra

Answer 3

Passing a list as the first argument with shell=True is system-dependent. I’m guessing you really mean

with open("%sCOSMOSIS_output_%s.txt" % (ERROR_PATH, ini), "ab") as file,
        open("%sCOSMOSIS_output_ERROR_%s.txt" % (ERROR_PATH, ini), "a") as file_e:
    try:
        job3 = subprocess.run(
            ["cosmosis", ini, arg2, COSMOSIS_PATH, os.path.splitext(ini)[0] + ".txt"],
            stdout=file, stderr=file_e, check=True,
            timeout=2000)
    except TimeoutExpired:
        pass

There is no way for job3.stdout or job3.stderr to contain anything because we redirected them to files. (We open the file handles with binary mode so we don’t need to specify a text encoding.) There is also no way for the process to return a useful result if it is killed by a timeout, and obviously no need to kill it when it was already killed.

As the subprocess documentation already tells you, you should prefer subprocess.run or one of the legacy high-level wrappers instead of Popen when you can. Perhaps see also Actual meaning of shell=True in subprocess which also explains why you want to avoid using a shell when you can.

On the other hand, if (as indicated in comments) you want the process to run in the background until completion while your Python script proceeds to perform other tasks and take out the timeout, you do need Popen.

with open("%sCOSMOSIS_output_%s.txt" % (ERROR_PATH, ini), "ab") as file,
        open("%sCOSMOSIS_output_ERROR_%s.txt" % (ERROR_PATH, ini), "a") as file_e:
    job3 = subprocess.Popen(
        ["cosmosis", ini, arg2, COSMOSIS_PATH, os.path.splitext(ini)[0] + ".txt"],
        stdout=file, stderr=file_e)
...
# time passes and your script does other things ...
job3.wait()

You might want to periodically poll the job to see if it has finished.

Answered By: tripleee

python subprocess does not write the output

Question:

Answers: