How to redirect output with subprocess in Python?
Question:
What I do in the command line:
cat file1 file2 file3 > myfile
What I want to do with python:
import subprocess, shlex
my_cmd = 'cat file1 file2 file3 > myfile'
args = shlex.split(my_cmd)
subprocess.call(args) # spits the output in the window i call my python program
Answers:
UPDATE: os.system is discouraged, albeit still available in Python 3.
Use os.system:
os.system(my_cmd)
If you really want to use subprocess, here’s the solution (mostly lifted from the documentation for subprocess):
p = subprocess.Popen(my_cmd, shell=True)
os.waitpid(p.pid, 0)
OTOH, you can avoid system calls entirely:
import shutil
with open('myfile', 'w') as outfile:
for infile in ('file1', 'file2', 'file3'):
shutil.copyfileobj(open(infile), outfile)
@PoltoS I want to join some files and then process the resulting file. I thought using cat was the easiest alternative. Is there a better/pythonic way to do it?
Of course:
with open('myfile', 'w') as outfile:
for infilename in ['file1', 'file2', 'file3']:
with open(infilename) as infile:
outfile.write(infile.read())
In Python 3.5+ to redirect the output, just pass an open file handle for the stdout argument to subprocess.run:
# Use a list of args instead of a string
input_files = ['file1', 'file2', 'file3']
my_cmd = ['cat'] + input_files
with open('myfile', "w") as outfile:
subprocess.run(my_cmd, stdout=outfile)
As others have pointed out, the use of an external command like cat for this purpose is completely extraneous.
One interesting case would be to update a file by appending similar file to it. Then one would not have to create a new file in the process. It is particularly useful in the case where a large file need to be appended. Here is one possibility using teminal command line directly from python.
import subprocess32 as sub
with open("A.csv","a") as f:
f.flush()
sub.Popen(["cat","temp.csv"],stdout=f)
size = 'ffprobe -v error -show_entries format=size -of default=noprint_wrappers=1:nokey=1 dump.mp4 > file'
proc = subprocess.Popen(shlex.split(size), shell=True)
time.sleep(1)
proc.terminate() #proc.kill() modify it by a suggestion
size = ""
with open('file', 'r') as infile:
for line in infile.readlines():
size += line.strip()
print(size)
os.remove('file')
When you use subprocess , the process must be killed.This is an example.If you don’t kill the process , file will be empty and you can read nothing.It can run on Windows.I can`t make sure that it can run on Unix.
It will work if your args will look like ['sh', '-c', 'cat file1 file2 file3 > myfile'] it will mean that output of cat won’t pass Python and spawn in shell instead (insted of sh -c you can use bash -c)
What I do in the command line:
cat file1 file2 file3 > myfile
What I want to do with python:
import subprocess, shlex
my_cmd = 'cat file1 file2 file3 > myfile'
args = shlex.split(my_cmd)
subprocess.call(args) # spits the output in the window i call my python program
UPDATE: os.system is discouraged, albeit still available in Python 3.
Use os.system:
os.system(my_cmd)
If you really want to use subprocess, here’s the solution (mostly lifted from the documentation for subprocess):
p = subprocess.Popen(my_cmd, shell=True)
os.waitpid(p.pid, 0)
OTOH, you can avoid system calls entirely:
import shutil
with open('myfile', 'w') as outfile:
for infile in ('file1', 'file2', 'file3'):
shutil.copyfileobj(open(infile), outfile)
@PoltoS I want to join some files and then process the resulting file. I thought using cat was the easiest alternative. Is there a better/pythonic way to do it?
Of course:
with open('myfile', 'w') as outfile:
for infilename in ['file1', 'file2', 'file3']:
with open(infilename) as infile:
outfile.write(infile.read())
In Python 3.5+ to redirect the output, just pass an open file handle for the stdout argument to subprocess.run:
# Use a list of args instead of a string
input_files = ['file1', 'file2', 'file3']
my_cmd = ['cat'] + input_files
with open('myfile', "w") as outfile:
subprocess.run(my_cmd, stdout=outfile)
As others have pointed out, the use of an external command like cat for this purpose is completely extraneous.
One interesting case would be to update a file by appending similar file to it. Then one would not have to create a new file in the process. It is particularly useful in the case where a large file need to be appended. Here is one possibility using teminal command line directly from python.
import subprocess32 as sub
with open("A.csv","a") as f:
f.flush()
sub.Popen(["cat","temp.csv"],stdout=f)
size = 'ffprobe -v error -show_entries format=size -of default=noprint_wrappers=1:nokey=1 dump.mp4 > file'
proc = subprocess.Popen(shlex.split(size), shell=True)
time.sleep(1)
proc.terminate() #proc.kill() modify it by a suggestion
size = ""
with open('file', 'r') as infile:
for line in infile.readlines():
size += line.strip()
print(size)
os.remove('file')
When you use subprocess , the process must be killed.This is an example.If you don’t kill the process , file will be empty and you can read nothing.It can run on Windows.I can`t make sure that it can run on Unix.
It will work if your args will look like ['sh', '-c', 'cat file1 file2 file3 > myfile'] it will mean that output of cat won’t pass Python and spawn in shell instead (insted of sh -c you can use bash -c)