Pickle incompatibility of numpy arrays between Python 2 and 3
Question:
I am trying to load the MNIST dataset linked here in Python 3.2 using this program:
import pickle
import gzip
import numpy
with gzip.open('mnist.pkl.gz', 'rb') as f:
l = list(pickle.load(f))
print(l)
Unfortunately, it gives me the error:
Traceback (most recent call last):
File "mnist.py", line 7, in <module>
train_set, valid_set, test_set = pickle.load(f)
UnicodeDecodeError: 'ascii' codec can't decode byte 0x90 in position 614: ordinal not in range(128)
I then tried to decode the pickled file in Python 2.7, and re-encode it. So, I ran this program in Python 2.7:
import pickle
import gzip
import numpy
with gzip.open('mnist.pkl.gz', 'rb') as f:
train_set, valid_set, test_set = pickle.load(f)
# Printing out the three objects reveals that they are
# all pairs containing numpy arrays.
with gzip.open('mnistx.pkl.gz', 'wb') as g:
pickle.dump(
(train_set, valid_set, test_set),
g,
protocol=2) # I also tried protocol 0.
It ran without error, so I reran this program in Python 3.2:
import pickle
import gzip
import numpy
# note the filename change
with gzip.open('mnistx.pkl.gz', 'rb') as f:
l = list(pickle.load(f))
print(l)
However, it gave me the same error as before. How do I get this to work?
Answers:
It looks like there are some compatablility issues in pickle between 2.x and 3.x due to the move to unicode. Your file appears to be pickled with python 2.x and decoding it in 3.x could be troublesome.
I’d suggest unpickling it with python 2.x and saving to a format that plays more nicely across the two versions you’re using.
This seems like some sort of incompatibility. It’s trying to load a “binstring” object, which is assumed to be ASCII, while in this case it is binary data. If this is a bug in the Python 3 unpickler, or a “misuse” of the pickler by numpy, I don’t know.
Here is something of a workaround, but I don’t know how meaningful the data is at this point:
import pickle
import gzip
import numpy
with open('mnist.pkl', 'rb') as f:
u = pickle._Unpickler(f)
u.encoding = 'latin1'
p = u.load()
print(p)
Unpickling it in Python 2 and then repickling it is only going to create the same problem again, so you need to save it in another format.
If you are getting this error in python3, then, it could be an incompatibility issue between python 2 and python 3, for me the solution was to load with latin1 encoding:
pickle.load(file, encoding='latin1')
It appears to be an incompatibility issue between Python 2 and Python 3. I tried loading the MNIST dataset with
train_set, valid_set, test_set = pickle.load(file, encoding='iso-8859-1')
and it worked for Python 3.5.2
I just stumbled upon this snippet. Hope this helps to clarify the compatibility issue.
import sys
with gzip.open('mnist.pkl.gz', 'rb') as f:
if sys.version_info.major > 2:
train_set, valid_set, test_set = pickle.load(f, encoding='latin1')
else:
train_set, valid_set, test_set = pickle.load(f)
There is hickle which is faster than pickle and easier.
I tried to save and read it in pickle dump but while reading there were a lot of problems and wasted an hour and still didn’t find a solution though I was working on my own data to create a chatbot.
vec_x and vec_y are numpy arrays:
data=[vec_x,vec_y]
hkl.dump( data, 'new_data_file.hkl' )
Then you just read it and perform the operations:
data2 = hkl.load( 'new_data_file.hkl' )
Try:
l = list(pickle.load(f, encoding='bytes')) #if you are loading image data or
l = list(pickle.load(f, encoding='latin1')) #if you are loading text data
From the documentation of pickle.load method:
Optional keyword arguments are fix_imports, encoding and errors, which are used to control compatibility support for pickle stream generated by Python 2.
If fix_imports is True, pickle will try to map the old Python 2 names to the new names used in Python 3.
The encoding and errors tell pickle how to decode 8-bit string instances pickled by Python 2; these default to ‘ASCII’ and ‘strict’, respectively. The encoding can be ‘bytes’ to read these 8-bit string instances as bytes objects.
I am trying to load the MNIST dataset linked here in Python 3.2 using this program:
import pickle
import gzip
import numpy
with gzip.open('mnist.pkl.gz', 'rb') as f:
l = list(pickle.load(f))
print(l)
Unfortunately, it gives me the error:
Traceback (most recent call last):
File "mnist.py", line 7, in <module>
train_set, valid_set, test_set = pickle.load(f)
UnicodeDecodeError: 'ascii' codec can't decode byte 0x90 in position 614: ordinal not in range(128)
I then tried to decode the pickled file in Python 2.7, and re-encode it. So, I ran this program in Python 2.7:
import pickle
import gzip
import numpy
with gzip.open('mnist.pkl.gz', 'rb') as f:
train_set, valid_set, test_set = pickle.load(f)
# Printing out the three objects reveals that they are
# all pairs containing numpy arrays.
with gzip.open('mnistx.pkl.gz', 'wb') as g:
pickle.dump(
(train_set, valid_set, test_set),
g,
protocol=2) # I also tried protocol 0.
It ran without error, so I reran this program in Python 3.2:
import pickle
import gzip
import numpy
# note the filename change
with gzip.open('mnistx.pkl.gz', 'rb') as f:
l = list(pickle.load(f))
print(l)
However, it gave me the same error as before. How do I get this to work?
It looks like there are some compatablility issues in pickle between 2.x and 3.x due to the move to unicode. Your file appears to be pickled with python 2.x and decoding it in 3.x could be troublesome.
I’d suggest unpickling it with python 2.x and saving to a format that plays more nicely across the two versions you’re using.
This seems like some sort of incompatibility. It’s trying to load a “binstring” object, which is assumed to be ASCII, while in this case it is binary data. If this is a bug in the Python 3 unpickler, or a “misuse” of the pickler by numpy, I don’t know.
Here is something of a workaround, but I don’t know how meaningful the data is at this point:
import pickle
import gzip
import numpy
with open('mnist.pkl', 'rb') as f:
u = pickle._Unpickler(f)
u.encoding = 'latin1'
p = u.load()
print(p)
Unpickling it in Python 2 and then repickling it is only going to create the same problem again, so you need to save it in another format.
If you are getting this error in python3, then, it could be an incompatibility issue between python 2 and python 3, for me the solution was to load with latin1 encoding:
pickle.load(file, encoding='latin1')
It appears to be an incompatibility issue between Python 2 and Python 3. I tried loading the MNIST dataset with
train_set, valid_set, test_set = pickle.load(file, encoding='iso-8859-1')
and it worked for Python 3.5.2
I just stumbled upon this snippet. Hope this helps to clarify the compatibility issue.
import sys
with gzip.open('mnist.pkl.gz', 'rb') as f:
if sys.version_info.major > 2:
train_set, valid_set, test_set = pickle.load(f, encoding='latin1')
else:
train_set, valid_set, test_set = pickle.load(f)
There is hickle which is faster than pickle and easier.
I tried to save and read it in pickle dump but while reading there were a lot of problems and wasted an hour and still didn’t find a solution though I was working on my own data to create a chatbot.
vec_x and vec_y are numpy arrays:
data=[vec_x,vec_y]
hkl.dump( data, 'new_data_file.hkl' )
Then you just read it and perform the operations:
data2 = hkl.load( 'new_data_file.hkl' )
Try:
l = list(pickle.load(f, encoding='bytes')) #if you are loading image data or
l = list(pickle.load(f, encoding='latin1')) #if you are loading text data
From the documentation of pickle.load method:
Optional keyword arguments are fix_imports, encoding and errors, which are used to control compatibility support for pickle stream generated by Python 2.
If fix_imports is True, pickle will try to map the old Python 2 names to the new names used in Python 3.
The encoding and errors tell pickle how to decode 8-bit string instances pickled by Python 2; these default to ‘ASCII’ and ‘strict’, respectively. The encoding can be ‘bytes’ to read these 8-bit string instances as bytes objects.