_csv.Error: field larger than field limit (131072)
Question:
I have a script reading in a csv file with very huge fields:
# example from http://docs.python.org/3.3/library/csv.html?highlight=csv%20dictreader#examples
import csv
with open('some.csv', newline='') as f:
reader = csv.reader(f)
for row in reader:
print(row)
However, this throws the following error on some csv files:
_csv.Error: field larger than field limit (131072)
How can I analyze csv files with huge fields? Skipping the lines with huge fields is not an option as the data needs to be analyzed in subsequent steps.
Answers:
The csv file might contain very huge fields, therefore increase the field_size_limit:
import sys
import csv
csv.field_size_limit(sys.maxsize)
sys.maxsize works for Python 2.x and 3.x. sys.maxint would only work with Python 2.x (SO: what-is-sys-maxint-in-python-3)
Update
As Geoff pointed out, the code above might result in the following error: OverflowError: Python int too large to convert to C long.
To circumvent this, you could use the following quick and dirty code (which should work on every system with Python 2 and Python 3):
import sys
import csv
maxInt = sys.maxsize
while True:
# decrease the maxInt value by factor 10
# as long as the OverflowError occurs.
try:
csv.field_size_limit(maxInt)
break
except OverflowError:
maxInt = int(maxInt/10)
This could be because your CSV file has embedded single or double quotes. If your CSV file is tab-delimited try opening it as:
c = csv.reader(f, delimiter='t', quoting=csv.QUOTE_NONE)
Sometimes, a row contain double quote column. When csv reader try read this row, not understood end of column and fire this raise.
Solution is below:
reader = csv.reader(cf, quoting=csv.QUOTE_MINIMAL)
Below is to check the current limit
csv.field_size_limit()
Out[20]: 131072
Below is to increase the limit. Add it to the code
csv.field_size_limit(100000000)
Try checking the limit again
csv.field_size_limit()
Out[22]: 100000000
Now you won’t get the error “_csv.Error: field larger than field limit (131072)”
Find the cqlshrc file usually placed in .cassandra directory.
In that file append,
[csv]
field_size_limit = 1000000000
.csv field sizes are controlled via [Python.Docs]: csv.field_size_limit([new_limit]) (emphasis is mine):
Returns the current maximum field size allowed by the parser. If new_limit is given, this becomes the new limit.
It is set by default to 131072 or 0x20000 (128k), which should be enough for any decent .csv:
>>> import csv
>>>
>>>
>>> limit0 = csv.field_size_limit()
>>> limit0
131072
>>> "0x{0:016X}".format(limit0)
'0x0000000000020000'
However, when dealing with a .csv file (with the correct quoting and delimiter) having (at least) one field longer than this size, the error pops up.
To get rid of the error, the size limit should be increased (to avoid any worries, the maximum possible value is attempted).
Behind the scenes (check [GitHub]: python/cpython – (master) cpython/Modules/_csv.c for implementation details), the variable that holds this value is a C long ([Wikipedia]: C data types), whose size varies depending on CPU architecture and OS (ILP). The classical difference: for a 064bit OS (and Python build), the long type size (in bits) is:
- Nix: 64
- Win: 32
When attempting to set it, the new value is checked to be in the long boundaries, that’s why in some cases another exception pops up (because sys.maxsize is typically 064bit wide – encountered on Win):
>>> import sys, ctypes as ct
>>>
>>>
>>> "v{:d}.{:d}.{:d}".format(*sys.version_info[:3]), sys.platform, sys.maxsize, ct.sizeof(ct.c_void_p) * 8, ct.sizeof(ct.c_long) * 8
('v3.9.9', 'win32', 9223372036854775807, 64, 32)
>>>
>>> csv.field_size_limit(sys.maxsize)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OverflowError: Python int too large to convert to C long
To avoid running into this problem, set the (maximum possible) limit (LONG_MAX), using an artifice (thanks to [Python.Docs]: ctypes – A foreign function library for Python). It should work on Python 3 and Python 2, on any CPU / OS.
>>> csv.field_size_limit(int(ct.c_ulong(-1).value // 2))
131072
>>> limit1 = csv.field_size_limit()
>>> limit1
2147483647
>>> "0x{0:016X}".format(limit1)
'0x000000007FFFFFFF'
064bit Python on a Nix like OS:
>>> import sys, csv, ctypes as ct
>>>
>>>
>>> "v{:d}.{:d}.{:d}".format(*sys.version_info[:3]), sys.platform, sys.maxsize, ct.sizeof(ct.c_void_p) * 8, ct.sizeof(ct.c_long) * 8
('v3.8.10', 'linux', 9223372036854775807, 64, 64)
>>>
>>> csv.field_size_limit()
131072
>>>
>>> csv.field_size_limit(int(ct.c_ulong(-1).value // 2))
131072
>>> limit1 = csv.field_size_limit()
>>> limit1
9223372036854775807
>>> "0x{0:016X}".format(limit1)
'0x7FFFFFFFFFFFFFFF'
For 032bit Python, things should run smoothly without the artifice (as both sys.maxsize and LONG_MAX are 032bit wide).
If this maximum value is still not enough, then the .csv would need manual intervention in order to be processed from Python.
Check the following resources for more details on:
-
Playing with C types boundaries from Python: [SO]: Maximum and minimum value of C types integers from Python (@CristiFati’s answer)
-
Python 064bit vs 032bit differences: [SO]: How do I determine if my python shell is executing in 32bit or 64bit mode on OS X? (@CristiFati’s answer)
I just had this happen to me on a ‘plain’ CSV file. Some people might call it an invalid formatted file. No escape characters, no double quotes and delimiter was a semicolon.
A sample line from this file would look like this:
First cell; Second ” Cell with one double quote and leading
space;’Partially quoted’ cell;Last cell
the single quote in the second cell would throw the parser off its rails. What worked was:
csv.reader(inputfile, delimiter=';', doublequote='False', quotechar='', quoting=csv.QUOTE_NONE)
You can use the error_bad_lines option of pd.read_csv to skip these lines.
import pandas as pd
data_df = pd.read_csv('data.csv', error_bad_lines=False)
This works since the "bad lines" as defined in pandas include lines that one of their fields exceed the csv limit.
Be careful that this solution is valid only when the fields in your csv file shouldn’t be this long.
If you expect to have big field sizes, this will throw away your data.
I have a script reading in a csv file with very huge fields:
# example from http://docs.python.org/3.3/library/csv.html?highlight=csv%20dictreader#examples
import csv
with open('some.csv', newline='') as f:
reader = csv.reader(f)
for row in reader:
print(row)
However, this throws the following error on some csv files:
_csv.Error: field larger than field limit (131072)
How can I analyze csv files with huge fields? Skipping the lines with huge fields is not an option as the data needs to be analyzed in subsequent steps.
The csv file might contain very huge fields, therefore increase the field_size_limit:
import sys
import csv
csv.field_size_limit(sys.maxsize)
sys.maxsize works for Python 2.x and 3.x. sys.maxint would only work with Python 2.x (SO: what-is-sys-maxint-in-python-3)
Update
As Geoff pointed out, the code above might result in the following error: OverflowError: Python int too large to convert to C long.
To circumvent this, you could use the following quick and dirty code (which should work on every system with Python 2 and Python 3):
import sys
import csv
maxInt = sys.maxsize
while True:
# decrease the maxInt value by factor 10
# as long as the OverflowError occurs.
try:
csv.field_size_limit(maxInt)
break
except OverflowError:
maxInt = int(maxInt/10)
This could be because your CSV file has embedded single or double quotes. If your CSV file is tab-delimited try opening it as:
c = csv.reader(f, delimiter='t', quoting=csv.QUOTE_NONE)
Sometimes, a row contain double quote column. When csv reader try read this row, not understood end of column and fire this raise.
Solution is below:
reader = csv.reader(cf, quoting=csv.QUOTE_MINIMAL)
Below is to check the current limit
csv.field_size_limit()
Out[20]: 131072
Below is to increase the limit. Add it to the code
csv.field_size_limit(100000000)
Try checking the limit again
csv.field_size_limit()
Out[22]: 100000000
Now you won’t get the error “_csv.Error: field larger than field limit (131072)”
Find the cqlshrc file usually placed in .cassandra directory.
In that file append,
[csv]
field_size_limit = 1000000000
.csv field sizes are controlled via [Python.Docs]: csv.field_size_limit([new_limit]) (emphasis is mine):
Returns the current maximum field size allowed by the parser. If new_limit is given, this becomes the new limit.
It is set by default to 131072 or 0x20000 (128k), which should be enough for any decent .csv:
>>> import csv >>> >>> >>> limit0 = csv.field_size_limit() >>> limit0 131072 >>> "0x{0:016X}".format(limit0) '0x0000000000020000'
However, when dealing with a .csv file (with the correct quoting and delimiter) having (at least) one field longer than this size, the error pops up.
To get rid of the error, the size limit should be increased (to avoid any worries, the maximum possible value is attempted).
Behind the scenes (check [GitHub]: python/cpython – (master) cpython/Modules/_csv.c for implementation details), the variable that holds this value is a C long ([Wikipedia]: C data types), whose size varies depending on CPU architecture and OS (ILP). The classical difference: for a 064bit OS (and Python build), the long type size (in bits) is:
- Nix: 64
- Win: 32
When attempting to set it, the new value is checked to be in the long boundaries, that’s why in some cases another exception pops up (because sys.maxsize is typically 064bit wide – encountered on Win):
>>> import sys, ctypes as ct >>> >>> >>> "v{:d}.{:d}.{:d}".format(*sys.version_info[:3]), sys.platform, sys.maxsize, ct.sizeof(ct.c_void_p) * 8, ct.sizeof(ct.c_long) * 8 ('v3.9.9', 'win32', 9223372036854775807, 64, 32) >>> >>> csv.field_size_limit(sys.maxsize) Traceback (most recent call last): File "<stdin>", line 1, in <module> OverflowError: Python int too large to convert to C long
To avoid running into this problem, set the (maximum possible) limit (LONG_MAX), using an artifice (thanks to [Python.Docs]: ctypes – A foreign function library for Python). It should work on Python 3 and Python 2, on any CPU / OS.
>>> csv.field_size_limit(int(ct.c_ulong(-1).value // 2)) 131072 >>> limit1 = csv.field_size_limit() >>> limit1 2147483647 >>> "0x{0:016X}".format(limit1) '0x000000007FFFFFFF'
064bit Python on a Nix like OS:
>>> import sys, csv, ctypes as ct >>> >>> >>> "v{:d}.{:d}.{:d}".format(*sys.version_info[:3]), sys.platform, sys.maxsize, ct.sizeof(ct.c_void_p) * 8, ct.sizeof(ct.c_long) * 8 ('v3.8.10', 'linux', 9223372036854775807, 64, 64) >>> >>> csv.field_size_limit() 131072 >>> >>> csv.field_size_limit(int(ct.c_ulong(-1).value // 2)) 131072 >>> limit1 = csv.field_size_limit() >>> limit1 9223372036854775807 >>> "0x{0:016X}".format(limit1) '0x7FFFFFFFFFFFFFFF'
For 032bit Python, things should run smoothly without the artifice (as both sys.maxsize and LONG_MAX are 032bit wide).
If this maximum value is still not enough, then the .csv would need manual intervention in order to be processed from Python.
Check the following resources for more details on:
-
Playing with C types boundaries from Python: [SO]: Maximum and minimum value of C types integers from Python (@CristiFati’s answer)
-
Python 064bit vs 032bit differences: [SO]: How do I determine if my python shell is executing in 32bit or 64bit mode on OS X? (@CristiFati’s answer)
I just had this happen to me on a ‘plain’ CSV file. Some people might call it an invalid formatted file. No escape characters, no double quotes and delimiter was a semicolon.
A sample line from this file would look like this:
First cell; Second ” Cell with one double quote and leading
space;’Partially quoted’ cell;Last cell
the single quote in the second cell would throw the parser off its rails. What worked was:
csv.reader(inputfile, delimiter=';', doublequote='False', quotechar='', quoting=csv.QUOTE_NONE)
You can use the error_bad_lines option of pd.read_csv to skip these lines.
import pandas as pd
data_df = pd.read_csv('data.csv', error_bad_lines=False)
This works since the "bad lines" as defined in pandas include lines that one of their fields exceed the csv limit.
Be careful that this solution is valid only when the fields in your csv file shouldn’t be this long.
If you expect to have big field sizes, this will throw away your data.