Switching to Python 3 causing UnicodeDecodeError

Question

I’ve just added Python3 interpreter to Sublime, and the following code stopped working:

for directory in directoryList:
    fileList = os.listdir(directory)
    for filename in fileList:
        filename = os.path.join(directory, filename)
        currentFile = open(filename, 'rt')
        for line in currentFile:               ##Here comes the exception.
            currentLine = line.split(' ')
            for word in currentLine:
                if word.lower() not in bigBagOfWords:
                    bigBagOfWords.append(word.lower())
        currentFile.close()

I get a following exception:

  File "/Users/Kuba/Desktop/DictionaryCreator.py", line 11, in <module>
    for line in currentFile:
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcc in position 305: ordinal not in range(128)

I found this rather strange, because as far as I know Python3 is supposed to support utf-8 everywhere. What’s more, the same exact code works with no problems on Python2.7. I’ve read about adding environmental variable PYTHONIOENCODING, but I tried it – to no avail (however, it appears it is not that easy to add an environmental variable in OS X Mavericks, so maybe I did something wrong with adding the variable? I modidified /etc/launchd.conf)

Asked By: 3yakuya

||

Source

Answer 1

Python 3 decodes text files when reading, encodes when writing. The default encoding is taken from locale.getpreferredencoding(False), which evidently for your setup returns 'ASCII'. See the open() function documenation:

In text mode, if encoding is not specified the encoding used is platform dependent: locale.getpreferredencoding(False) is called to get the current locale encoding.

Instead of relying on a system setting, you should open your text files using an explicit codec:

currentFile = open(filename, 'rt', encoding='latin1')

where you set the encoding parameter to match the file you are reading.

Python 3 supports UTF-8 as the default for source code.

The same applies to writing to a writeable text file; data written will be encoded, and if you rely on the system encoding you are liable to get UnicodeEncodingError exceptions unless you explicitly set a suitable codec. What codec to use when writing depends on what text you are writing and what you plan to do with the file afterward.

You may want to read up on Python 3 and Unicode in the Unicode HOWTO, which explains both about source code encoding and reading and writing Unicode data.

Answered By: Martijn Pieters

Answer 2

“as far as I know Python3 is supposed to support utf-8 everywhere …”
Not true. I have python 3.6 and my default encoding is NOT utf-8.
To change it to utf-8 in my code I use:

import locale
def getpreferredencoding(do_setlocale = True):
   return "utf-8"
locale.getpreferredencoding = getpreferredencoding

as explained in
Changing the “locale preferred encoding” in Python 3 in Windows

Answered By: Farid Khafizov

Answer 3

In general, I found 3 ways to fix Unicode related Errors in Python3:

Use the encoding explicitly like currentFile = open(filename, ‘rt’,encoding=’utf-8′)
As the bytes have no encoding, convert the string data to bytes before writing to file like data = ‘string’.encode(‘utf-8’)
Especially in Linux environment, check $LANG. Such issue usually arises when LANG=C which makes default encoding as ‘ascii’ instead of ‘utf-8′. One can change it with other appropriate value like LANG=’en_IN’

Answered By: vicky_kqr

Switching to Python 3 causing UnicodeDecodeError

Question:

Answers: