IPython: redirecting output of a Python script to a file (like bash >)
Question:
I have a Python script that I want to run in IPython. I want to redirect (write) the output to a file, similar to:
python my_script.py > my_output.txt
How do I do this when I run the script in IPython, i.e. like execfile('my_script.py')
There is an older page describing a function that could be written to do this, but I believe that there is now a built-in way to do this that I just can’t find.
Answers:
There’s the hacky way of overwriting sys.stdout and sys.stderr with a file object, but that’s really not a good way to go about it. Really, if you want to control where the output goes from inside python, you need to implement some sort of logging and/or output handling system that you can configure via the command line or function arguments instead of using print statements.
IPython has its own context manager for capturing stdout/err, but it doesn’t redirect to files, it redirects to an object:
from IPython.utils import io
with io.capture_output() as captured:
%run my_script.py
print captured.stdout # prints stdout from your script
And this functionality is exposed in a %%capture cell-magic, as illustrated in the Cell Magics example notebook.
It’s a simple context manager, so you can write your own version that would redirect to files:
class redirect_output(object):
"""context manager for reditrecting stdout/err to files"""
def __init__(self, stdout='', stderr=''):
self.stdout = stdout
self.stderr = stderr
def __enter__(self):
self.sys_stdout = sys.stdout
self.sys_stderr = sys.stderr
if self.stdout:
sys.stdout = open(self.stdout, 'w')
if self.stderr:
if self.stderr == self.stdout:
sys.stderr = sys.stdout
else:
sys.stderr = open(self.stderr, 'w')
def __exit__(self, exc_type, exc_value, traceback):
sys.stdout = self.sys_stdout
sys.stderr = self.sys_stderr
which you would invoke with:
with redirect_output("my_output.txt"):
%run my_script.py
It seems a lot of code….
My solution.
redirect output of ipython script into a csv or text file like sqlplus spool
wonder there is an easy way like oracle sqlplus spool command..?
To quickly store text contained in a variable while working in IPython use %store with > or >>:
%store VARIABLE >>file.txt (appends)
%store VARIABLE >file.txt (overwrites)
(Make sure there is no space immediately following the > or >>)
While this an old question, I found this and the answers as I was facing a similar problem.
The solution I found after sifting through IPython Cell magics documentation is actually fairly simple. At the most basic the solution is to assign the output of the command to a variable.
This simple two-cell example shows how to do that. In the first Notebook cell we define the Python script with some output to stdout making use of the %%writefile cell magic.
%%writefile output.py
print("This is the output that is supposed to go to a file")
Then we run that script like it was run from a shell using the ! operator.
output = !python output.py
print(output)
>>> ['This is the output that is supposed to go to a file']
Then you can easily make use of the %store magic to persist the output.
%store output >output.log
Notice however that the output of the command is persisted as a list of lines. You might want to call "n".join(output) prior storing the output.
use this code to save the output to file
import time
from threading import Thread
import sys
#write the stdout to file
def log():
#for stop the thread
global run
while (run):
try:
global out
text = str(sys.stdout.getvalue())
with open("out.txt", 'w') as f:
f.write(text)
finally:
time.sleep(1)
%%capture out
run = True
print("start")
process = Thread(target=log, args=[]).start()
# do some work
for i in range(10, 1000):
print(i)
time.sleep(1)
run= False
process.join()
It is useful to use a text editor that tracer changes the file and suggest reloading the file like
notepad++
For just one script to run I would do the redirection in bash
ipython -c "execfile('my_script.py')" > my_output.txt
On python 3, execfile does not exist any more, so use this instead
ipython -c "exec(open('my_script.py').read())" > my_output.txt
Be careful with the double vs single quotes.
I wonder why the verified solution doesn’t entirely work in a loop, the following:
for i in range(N):
with redirect_output("path_to_output_file"):
%run <python_script> arg1 arg2 arg3
creates N files in the directory with only output from the first print statement of the <python_script>. Just confirming- when run separately for each iteration of the for loop, the script produces the right result.
I have a Python script that I want to run in IPython. I want to redirect (write) the output to a file, similar to:
python my_script.py > my_output.txt
How do I do this when I run the script in IPython, i.e. like execfile('my_script.py')
There is an older page describing a function that could be written to do this, but I believe that there is now a built-in way to do this that I just can’t find.
There’s the hacky way of overwriting sys.stdout and sys.stderr with a file object, but that’s really not a good way to go about it. Really, if you want to control where the output goes from inside python, you need to implement some sort of logging and/or output handling system that you can configure via the command line or function arguments instead of using print statements.
IPython has its own context manager for capturing stdout/err, but it doesn’t redirect to files, it redirects to an object:
from IPython.utils import io
with io.capture_output() as captured:
%run my_script.py
print captured.stdout # prints stdout from your script
And this functionality is exposed in a %%capture cell-magic, as illustrated in the Cell Magics example notebook.
It’s a simple context manager, so you can write your own version that would redirect to files:
class redirect_output(object):
"""context manager for reditrecting stdout/err to files"""
def __init__(self, stdout='', stderr=''):
self.stdout = stdout
self.stderr = stderr
def __enter__(self):
self.sys_stdout = sys.stdout
self.sys_stderr = sys.stderr
if self.stdout:
sys.stdout = open(self.stdout, 'w')
if self.stderr:
if self.stderr == self.stdout:
sys.stderr = sys.stdout
else:
sys.stderr = open(self.stderr, 'w')
def __exit__(self, exc_type, exc_value, traceback):
sys.stdout = self.sys_stdout
sys.stderr = self.sys_stderr
which you would invoke with:
with redirect_output("my_output.txt"):
%run my_script.py
It seems a lot of code….
My solution.
redirect output of ipython script into a csv or text file like sqlplus spool
wonder there is an easy way like oracle sqlplus spool command..?
To quickly store text contained in a variable while working in IPython use %store with > or >>:
%store VARIABLE >>file.txt (appends)
%store VARIABLE >file.txt (overwrites)
(Make sure there is no space immediately following the > or >>)
While this an old question, I found this and the answers as I was facing a similar problem.
The solution I found after sifting through IPython Cell magics documentation is actually fairly simple. At the most basic the solution is to assign the output of the command to a variable.
This simple two-cell example shows how to do that. In the first Notebook cell we define the Python script with some output to stdout making use of the %%writefile cell magic.
%%writefile output.py
print("This is the output that is supposed to go to a file")
Then we run that script like it was run from a shell using the ! operator.
output = !python output.py
print(output)
>>> ['This is the output that is supposed to go to a file']
Then you can easily make use of the %store magic to persist the output.
%store output >output.log
Notice however that the output of the command is persisted as a list of lines. You might want to call "n".join(output) prior storing the output.
use this code to save the output to file
import time
from threading import Thread
import sys
#write the stdout to file
def log():
#for stop the thread
global run
while (run):
try:
global out
text = str(sys.stdout.getvalue())
with open("out.txt", 'w') as f:
f.write(text)
finally:
time.sleep(1)
%%capture out
run = True
print("start")
process = Thread(target=log, args=[]).start()
# do some work
for i in range(10, 1000):
print(i)
time.sleep(1)
run= False
process.join()
It is useful to use a text editor that tracer changes the file and suggest reloading the file like
notepad++
For just one script to run I would do the redirection in bash
ipython -c "execfile('my_script.py')" > my_output.txt
On python 3, execfile does not exist any more, so use this instead
ipython -c "exec(open('my_script.py').read())" > my_output.txt
Be careful with the double vs single quotes.
I wonder why the verified solution doesn’t entirely work in a loop, the following:
for i in range(N):
with redirect_output("path_to_output_file"):
%run <python_script> arg1 arg2 arg3
creates N files in the directory with only output from the first print statement of the <python_script>. Just confirming- when run separately for each iteration of the for loop, the script produces the right result.