Can I redirect unicode output from the console directly into a file?
Question:
I’ve got a python script that outputs unicode to the console, and I’d like to redirect it to a file. Apparently, the redirect process in python involves converting the output to a string, so I get errors about inability to decode unicode characters.
So then, is there any way to perform a redirect into a file encoded in UTF-8?
Answers:
import codecs
file_object = codecs.open( "filename", "w", "utf-8" )
file_object.write(u"खऔणन")
file_object.close()
This should do the job.
When printing to the console, Python looks at sys.stdout.encoding to determine the encoding to use to encode unicode objects before printing.
When redirecting output to a file, sys.stdout.encoding is None, so Python2 defaults to the ascii encoding. (In contrast, Python3 defaults to utf-8.) This often leads to an exception when printing unicode.
You can avoid the error by explicitly encoding the unicode yourself before printing:
print (unicode_obj.encode('utf-8'))
or you could redefine sys.stdout so all output is encoded in utf-8:
import sys
import codecs
sys.stdout=codecs.getwriter('utf-8')(sys.stdout)
print(unicode_obj)
Set the environment variable PYTHONIOENCODING to the encoding you want before redirecting a python script to a file. Then you won’t have to modify the original script. Make sure to write Unicode strings as well, otherwise PYTHONIOENCODING will have no effect. If you write byte strings, the bytes are sent as-is to the terminal (or redirected file).
Under Linux, you can use tee and redirect stderr to /dev/null.
python script.py 2>/dev/null | tee filename.txt
You also don’t need to modify your Python script.
I’ve got a python script that outputs unicode to the console, and I’d like to redirect it to a file. Apparently, the redirect process in python involves converting the output to a string, so I get errors about inability to decode unicode characters.
So then, is there any way to perform a redirect into a file encoded in UTF-8?
import codecs
file_object = codecs.open( "filename", "w", "utf-8" )
file_object.write(u"खऔणन")
file_object.close()
This should do the job.
When printing to the console, Python looks at sys.stdout.encoding to determine the encoding to use to encode unicode objects before printing.
When redirecting output to a file, sys.stdout.encoding is None, so Python2 defaults to the ascii encoding. (In contrast, Python3 defaults to utf-8.) This often leads to an exception when printing unicode.
You can avoid the error by explicitly encoding the unicode yourself before printing:
print (unicode_obj.encode('utf-8'))
or you could redefine sys.stdout so all output is encoded in utf-8:
import sys
import codecs
sys.stdout=codecs.getwriter('utf-8')(sys.stdout)
print(unicode_obj)
Set the environment variable PYTHONIOENCODING to the encoding you want before redirecting a python script to a file. Then you won’t have to modify the original script. Make sure to write Unicode strings as well, otherwise PYTHONIOENCODING will have no effect. If you write byte strings, the bytes are sent as-is to the terminal (or redirected file).
Under Linux, you can use tee and redirect stderr to /dev/null.
python script.py 2>/dev/null | tee filename.txt
You also don’t need to modify your Python script.