UnicodeDecodeError when reading CSV file in Pandas with Python

Question:

I’m running a program which is processing 30,000 similar files. A random number of them are stopping and producing this error…

File "C:Importersrcdfmanimporter.py", line 26, in import_chr
     data = pd.read_csv(filepath, names=fields)
File "C:Python33libsite-packagespandasioparsers.py", line 400, in parser_f
     return _read(filepath_or_buffer, kwds)
File "C:Python33libsite-packagespandasioparsers.py", line 205, in _read
     return parser.read()
   File "C:Python33libsite-packagespandasioparsers.py", line 608, in read
     ret = self._engine.read(nrows)
File "C:Python33libsite-packagespandasioparsers.py", line 1028, in read
     data = self._reader.read(nrows)
File "parser.pyx", line 706, in pandas.parser.TextReader.read (pandasparser.c:6745)
File "parser.pyx", line 728, in pandas.parser.TextReader._read_low_memory (pandasparser.c:6964)
File "parser.pyx", line 804, in pandas.parser.TextReader._read_rows (pandasparser.c:7780)
File "parser.pyx", line 890, in pandas.parser.TextReader._convert_column_data (pandasparser.c:8793)
File "parser.pyx", line 950, in pandas.parser.TextReader._convert_tokens (pandasparser.c:9484)
File "parser.pyx", line 1026, in pandas.parser.TextReader._convert_with_dtype (pandasparser.c:10642)
File "parser.pyx", line 1046, in pandas.parser.TextReader._string_convert (pandasparser.c:10853)
File "parser.pyx", line 1278, in pandas.parser._string_box_utf8 (pandasparser.c:15657)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xda in position 6: invalid    continuation byte

The source/creation of these files all come from the same place. What’s the best way to correct this to proceed with the import?

Asked By: TravisVOX

||

Answers:

read_csv takes an encoding option to deal with files in different formats. I mostly use read_csv('file', encoding = "ISO-8859-1"), or alternatively encoding = "utf-8" for reading, and generally utf-8 for to_csv.

You can also use one of several alias options like 'latin' or 'cp1252' (Windows) instead of 'ISO-8859-1' (see python docs, also for numerous other encodings you may encounter).

See relevant Pandas documentation,
python docs examples on csv files, and plenty of related questions here on SO. A good background resource is What every developer should know about unicode and character sets.

To detect the encoding (assuming the file contains non-ascii characters), you can use enca (see man page) or file -i (linux) or file -I (osx) (see man page).

Answered By: Stefan

Simplest of all Solutions:

import pandas as pd
df = pd.read_csv('file_name.csv', engine='python')

Alternate Solution:

Sublime Text:

  • Open the csv file in Sublime text editor or VS Code.
  • Save the file in utf-8 format.
  • In sublime, Click File -> Save with encoding -> UTF-8

VS Code:

In the bottom bar of VSCode, you’ll see the label UTF-8. Click it. A popup opens. Click Save with encoding. You can now pick a new encoding for that file.

Then, you could read your file as usual:

import pandas as pd
data = pd.read_csv('file_name.csv', encoding='utf-8')

and the other different encoding types are:

encoding = "cp1252"
encoding = "ISO-8859-1"
Answered By: Gil Baggio

Struggled with this a while and thought I’d post on this question as it’s the first search result. Adding the encoding="iso-8859-1" tag to pandas read_csv didn’t work, nor did any other encoding, kept giving a UnicodeDecodeError.

If you’re passing a file handle to pd.read_csv(), you need to put the encoding attribute on the file open, not in read_csv. Obvious in hindsight, but a subtle error to track down.

Answered By: J. Ternent

Pandas allows to specify encoding, but does not allow to ignore errors not to automatically replace the offending bytes. So there is no one size fits all method but different ways depending on the actual use case.

  1. You know the encoding, and there is no encoding error in the file.
    Great: you have just to specify the encoding:

    file_encoding = 'cp1252'        # set file_encoding to the file encoding (utf8, latin1, etc.)
    pd.read_csv(input_file_and_path, ..., encoding=file_encoding)
    
  2. You do not want to be bothered with encoding questions, and only want that damn file to load, no matter if some text fields contain garbage. Ok, you only have to use Latin1 encoding because it accept any possible byte as input (and convert it to the unicode character of same code):

    pd.read_csv(input_file_and_path, ..., encoding='latin1')
    
  3. You know that most of the file is written with a specific encoding, but it also contains encoding errors. A real world example is an UTF8 file that has been edited with a non utf8 editor and which contains some lines with a different encoding. Pandas has no provision for a special error processing, but Python open function has (assuming Python3), and read_csv accepts a file like object. Typical errors parameter to use here are 'ignore' which just suppresses the offending bytes or (IMHO better) 'backslashreplace' which replaces the offending bytes by their Python’s backslashed escape sequence:

    file_encoding = 'utf8'        # set file_encoding to the file encoding (utf8, latin1, etc.)
    input_fd = open(input_file_and_path, encoding=file_encoding, errors = 'backslashreplace')
    pd.read_csv(input_fd, ...)
    
Answered By: Serge Ballesta
with open('filename.csv') as f:
   print(f)

after executing this code you will find encoding of ‘filename.csv’ then execute code as following

data=pd.read_csv('filename.csv', encoding="encoding as you found earlier"

there you go

Answered By: bhavesh

This answer seems to be the catch-all for CSV encoding issues. If you are getting a strange encoding problem with your header like this:

>>> f = open(filename,"r")
>>> reader = DictReader(f)
>>> next(reader)
OrderedDict([('ufeffid', '1'), ... ])

Then you have a byte order mark (BOM) character at the beginning of your CSV file. This answer addresses the issue:

Python read csv – BOM embedded into the first key

The solution is to load the CSV with encoding="utf-8-sig":

>>> f = open(filename,"r", encoding="utf-8-sig")
>>> reader = DictReader(f)
>>> next(reader)
OrderedDict([('id', '1'), ... ])

Hopefully this helps someone.

Answered By: nbwoodward

In my case, a file has USC-2 LE BOM encoding, according to Notepad++.
It is encoding="utf_16_le" for python.

Hope, it helps to find an answer a bit faster for someone.

Answered By: Vodyanikov Andrew

I am posting an update to this old thread. I found one solution that worked, but requires opening each file. I opened my csv file in LibreOffice, chose Save As > edit filter settings. In the drop-down menu I chose UTF8 encoding. Then I added encoding="utf-8-sig" to the data = pd.read_csv(r'C:fullpathtofilefilename.csv', sep = ',', encoding="utf-8-sig").

Hope this helps someone.

Answered By: tshirtdr1

Try specifying the engine=’python’.
It worked for me but I’m still trying to figure out why.

df = pd.read_csv(input_file_path,...engine='python')
Answered By: Jan33

I am using Jupyter-notebook. And in my case, it was showing the file in the wrong format. The ‘encoding’ option was not working.
So I save the csv in utf-8 format, and it works.

Answered By: Himanshu Sharma

In my case this worked for python 2.7:

data = read_csv(filename, encoding = "ISO-8859-1", dtype={'name_of_colum': unicode}, low_memory=False) 

And for python 3, only:

data = read_csv(filename, encoding = "ISO-8859-1", low_memory=False) 
Answered By: Victor Villacorta

Try this:

import pandas as pd
with open('filename.csv') as f:
    data = pd.read_csv(f)

Looks like it will take care of the encoding without explicitly expressing it through argument

Answered By: Ke Xu

Check the encoding before you pass to pandas. It will slow you down, but…

with open(path, 'r') as f:
    encoding = f.encoding 

df = pd.read_csv(path,sep=sep, encoding=encoding)

In python 3.7

Answered By: DaveP

I am posting an answer to provide an updated solution and explanation as to why this problem can occur. Say you are getting this data from a database or Excel workbook. If you have special characters like La Cañada Flintridge city, well unless you are exporting the data using UTF-8 encoding, you’re going to introduce errors. La Cañada Flintridge city will become La Caxf1ada Flintridge city. If you are using pandas.read_csv without any adjustments to the default parameters, you’ll hit the following error

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 5: invalid continuation byte

Fortunately, there are a few solutions.

Option 1, fix the exporting. Be sure to use UTF-8 encoding.

Option 2, if fixing the exporting problem is not available to you, and you need to use pandas.read_csv, be sure to include the following paramters, engine='python'. By default, pandas uses engine='C' which is great for reading large clean files, but will crash if anything unexpected comes up. In my experience, setting encoding='utf-8' has never fixed this UnicodeDecodeError. Also, you do not need to use errors_bad_lines, however, that is still an option if you REALLY need it.

pd.read_csv(<your file>, engine='python')

Option 3: solution is my preferred solution personally. Read the file using vanilla Python.

import pandas as pd

data = []

with open(<your file>, "rb") as myfile:
    # read the header seperately
    # decode it as 'utf-8', remove any special characters, and split it on the comma (or deliminator)
    header = myfile.readline().decode('utf-8').replace('rn', '').split(',')
    # read the rest of the data
    for line in myfile:
        row = line.decode('utf-8', errors='ignore').replace('rn', '').split(',')
        data.append(row)

# save the data as a dataframe
df = pd.DataFrame(data=data, columns = header)

Hope this helps people encountering this issue for the first time.

Answered By: Jon

I have trouble opening a CSV file in simplified Chinese downloaded from an online bank,
I have tried latin1, I have tried iso-8859-1, I have tried cp1252, all to no avail.

But pd.read_csv("",encoding ='gbk') simply does the work.

Answered By: Luk Aron

Another important issue that I faced which resulted in the same error was:

_values = pd.read_csv("C:UsersMujeebDesktopfile.xlxs")

^This line resulted in the same error because I am reading an excel file using read_csv() method. Use read_excel() for reading .xlxs

Answered By: Mujeeb Ishaque

Please try to add

import pandas as pd
df = pd.read_csv('file.csv', encoding='unicode_escape')

This will help. Worked for me. Also, make sure you’re using the correct delimiter and column names.

You can start with loading just 1000 rows to load the file quickly.

Answered By: Prakhar Rathi

You can try this.

import csv
import pandas as pd
df = pd.read_csv(filepath,encoding='unicode_escape')
Answered By: Dileep Dominic

You can try with:

df = pd.read_csv('./file_name.csv', encoding='gbk')

Answered By: ah bon

Try changing the encoding.
In my case, encoding = "utf-16" worked.

df = pd.read_csv("file.csv",encoding='utf-16')

Answered By: Kunal Kumar

Sometimes the problem is with the .csv file only. The file may be corrupted.
When faced with this issue. ‘Save As’ the file as csv again.

0. Open the xls/csv file
1. Go to -> files 
2. Click -> Save As 
3. Write the file name 
4. Choose 'file type' as -> CSV [very important]
5. Click -> Ok 
Answered By: Om Rastogi

This is a more general script approach for the stated question.

import pandas as pd

encoding_list = ['ascii', 'big5', 'big5hkscs', 'cp037', 'cp273', 'cp424', 'cp437', 'cp500', 'cp720', 'cp737'
                 , 'cp775', 'cp850', 'cp852', 'cp855', 'cp856', 'cp857', 'cp858', 'cp860', 'cp861', 'cp862'
                 , 'cp863', 'cp864', 'cp865', 'cp866', 'cp869', 'cp874', 'cp875', 'cp932', 'cp949', 'cp950'
                 , 'cp1006', 'cp1026', 'cp1125', 'cp1140', 'cp1250', 'cp1251', 'cp1252', 'cp1253', 'cp1254'
                 , 'cp1255', 'cp1256', 'cp1257', 'cp1258', 'euc_jp', 'euc_jis_2004', 'euc_jisx0213', 'euc_kr'
                 , 'gb2312', 'gbk', 'gb18030', 'hz', 'iso2022_jp', 'iso2022_jp_1', 'iso2022_jp_2'
                 , 'iso2022_jp_2004', 'iso2022_jp_3', 'iso2022_jp_ext', 'iso2022_kr', 'latin_1', 'iso8859_2'
                 , 'iso8859_3', 'iso8859_4', 'iso8859_5', 'iso8859_6', 'iso8859_7', 'iso8859_8', 'iso8859_9'
                 , 'iso8859_10', 'iso8859_11', 'iso8859_13', 'iso8859_14', 'iso8859_15', 'iso8859_16', 'johab'
                 , 'koi8_r', 'koi8_t', 'koi8_u', 'kz1048', 'mac_cyrillic', 'mac_greek', 'mac_iceland', 'mac_latin2'
                 , 'mac_roman', 'mac_turkish', 'ptcp154', 'shift_jis', 'shift_jis_2004', 'shift_jisx0213', 'utf_32'
                 , 'utf_32_be', 'utf_32_le', 'utf_16', 'utf_16_be', 'utf_16_le', 'utf_7', 'utf_8', 'utf_8_sig']

for encoding in encoding_list:
    worked = True
    try:
        df = pd.read_csv(path, encoding=encoding, nrows=5)
    except:
        worked = False
    if worked:
        print(encoding, ':n', df.head())

One starts with all the standard encodings available for the python version (in this case 3.7 python 3.7 standard encodings).
A usable python list of the standard encodings for the different python version is provided here: Helpful Stack overflow answer

Trying each encoding on a small chunk of the data;
only printing the working encoding.
The output is directly obvious.
This output also addresses the problem that an encoding like ‘latin1’ that runs through with ought any error, does not necessarily produce the wanted outcome.

In case of the question, I would try this approach specific for problematic CSV file and then maybe try to use the found working encoding for all others.

Answered By: Fledias weh

Pandas does not automatically replace the offending bytes by changing the encoding style. In my case, changing the encoding parameter from encoding = "utf-8" to encoding = "utf-16" resolved the issue.

Answered By: Reza Rahimi

In my case, I could not manage to overcome this issue using any method provided before. Changing the encoder type to utf-8, utf-16, iso-8859-1, or any other type somehow did not work.

But instead of using pd.read_csv(filename, delimiter=';'), I used;

pd.read_csv(open(filename, 'r'), delimiter=';')

and things seem working just fine.

Answered By: Onur Kirman

You can always try to detect the encoding of the file first, with chardet or cchardet:

from pathlib import Path
import chardet

filename = "file_name.csv"
detected = chardet.detect(Path(filename).read_bytes())
# detected is something like {'encoding': 'utf-8', 'confidence': 0.99, 'language': ''}

encoding = detected.get("encoding")
assert encoding, "Unable to detect encoding, is it a binary file?"

df = pd.read_csv(filename, encoding=encoding)

Answered By: mikey