How can I use pickle to save a dict (or any other Python object)?
Question:
I have looked through the information that the Python docs give, but I’m still a little confused. Could somebody post sample code that would write a new file then use pickle to dump a dictionary into it?
Answers:
Try this:
import pickle
a = {'hello': 'world'}
with open('filename.pickle', 'wb') as handle:
pickle.dump(a, handle, protocol=pickle.HIGHEST_PROTOCOL)
with open('filename.pickle', 'rb') as handle:
b = pickle.load(handle)
print(a == b)
There’s nothing about the above solution that is specific to a dict object. This same approach will will work for many Python objects, including instances of arbitrary classes and arbitrarily complex nestings of data structures. For example, replacing the second line with these lines:
import datetime
today = datetime.datetime.now()
a = [{'hello': 'world'}, 1, 2.3333, 4, True, "x",
("y", [[["z"], "y"], "x"]), {'today', today}]
will produce a result of True as well.
Some objects can’t be pickled due to their very nature. For example, it doesn’t make sense to pickle a structure containing a handle to an open file.
>>> import pickle
>>> with open("/tmp/picklefile", "wb") as f:
... pickle.dump({}, f)
...
normally it’s preferable to use the cPickle implementation
>>> import cPickle as pickle
>>> help(pickle.dump)
Help on built-in function dump in module cPickle:
dump(...)
dump(obj, file, protocol=0) -- Write an object in pickle format to the given file.
See the Pickler docstring for the meaning of optional argument proto.
import pickle
dictobj = {'Jack' : 123, 'John' : 456}
filename = "/foldername/filestore"
fileobj = open(filename, 'wb')
pickle.dump(dictobj, fileobj)
fileobj.close()
# Save a dictionary into a pickle file.
import pickle
favorite_color = {"lion": "yellow", "kitty": "red"} # create a dictionary
pickle.dump(favorite_color, open("save.p", "wb")) # save it into a file named save.p
# -------------------------------------------------------------
# Load the dictionary back from the pickle file.
import pickle
favorite_color = pickle.load(open("save.p", "rb"))
# favorite_color is now {"lion": "yellow", "kitty": "red"}
I’ve found pickling confusing (possibly because I’m thick). I found that this works, though:
myDictionaryString=str(myDictionary)
Which you can then write to a text file. I gave up trying to use pickle as I was getting errors telling me to write integers to a .dat file. I apologise for not using pickle.
In general, pickling a dict will fail unless you have only simple objects in it, like strings and integers.
Python 2.7.9 (default, Dec 11 2014, 01:21:43)
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from numpy import *
>>> type(globals())
<type 'dict'>
>>> import pickle
>>> pik = pickle.dumps(globals())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
Pickler(file, protocol).dump(obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
save(v)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 306, in save
rv = reduce(self.proto)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle module objects
>>>
Even a really simple dict will often fail. It just depends on the contents.
>>> d = {'x': lambda x:x}
>>> pik = pickle.dumps(d)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
Pickler(file, protocol).dump(obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
save(v)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 748, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle <function <lambda> at 0x102178668>: it's not found as __main__.<lambda>
However, if you use a better serializer like dill or cloudpickle, then most dictionaries can be pickled:
>>> import dill
>>> pik = dill.dumps(d)
Or if you want to save your dict to a file…
>>> with open('save.pik', 'w') as f:
... dill.dump(globals(), f)
...
The latter example is identical to any of the other good answers posted here (which aside from neglecting the picklability of the contents of the dict are good).
import pickle
your_data = {'foo': 'bar'}
# Store data (serialize)
with open('filename.pickle', 'wb') as handle:
pickle.dump(your_data, handle, protocol=pickle.HIGHEST_PROTOCOL)
# Load data (deserialize)
with open('filename.pickle', 'rb') as handle:
unserialized_data = pickle.load(handle)
print(your_data == unserialized_data)
The advantage of HIGHEST_PROTOCOL is that files get smaller. This makes unpickling sometimes much faster.
Important notice: The maximum file size of pickle is about 2GB.
Alternative way
import mpu
your_data = {'foo': 'bar'}
mpu.io.write('filename.pickle', data)
unserialized_data = mpu.io.read('filename.pickle')
Alternative Formats
- CSV: Super simple format (read & write)
- JSON: Nice for writing human-readable data; VERY commonly used (read & write)
- YAML: YAML is a superset of JSON, but easier to read (read & write, comparison of JSON and YAML)
- pickle: A Python serialization format (read & write)
- MessagePack (Python package): More compact representation (read & write)
- HDF5 (Python package): Nice for matrices (read & write)
- XML: exists too *sigh* (read & write)
For your application, the following might be important:
- Support by other programming languages
- Reading / writing performance
- Compactness (file size)
See also: Comparison of data serialization formats
In case you are rather looking for a way to make configuration files, you might want to read my short article Configuration files in Python
If you just want to store the dict in a single file, use pickle like that
import pickle
a = {'hello': 'world'}
with open('filename.pickle', 'wb') as handle:
pickle.dump(a, handle)
with open('filename.pickle', 'rb') as handle:
b = pickle.load(handle)
If you want to save and restore multiple dictionaries in multiple files for
caching and store more complex data,
use anycache.
It does all the other stuff you need around pickle
from anycache import anycache
@anycache(cachedir='path/to/files')
def myfunc(hello):
return {'hello', hello}
Anycache stores the different myfunc results depending on the arguments to
different files in cachedir and reloads them.
See the documentation for any further details.
Simple way to dump a Python data (e.g. dictionary) to a pickle file.
import pickle
your_dictionary = {}
pickle.dump(your_dictionary, open('pickle_file_name.p', 'wb'))
FYI, Pandas has a method to save pickles now.
I find it easier.
pd.to_pickle(object_to_save,'/temp/saved_pkl.pickle' )
If you want to handle writing or reading in one line without file opening:
import joblib
my_dict = {'hello': 'world'}
joblib.dump(my_dict, "my_dict.pickle") # write pickle file
my_dict_loaded = joblib.load("my_dict.pickle") # read pickle file
I have looked through the information that the Python docs give, but I’m still a little confused. Could somebody post sample code that would write a new file then use pickle to dump a dictionary into it?
Try this:
import pickle
a = {'hello': 'world'}
with open('filename.pickle', 'wb') as handle:
pickle.dump(a, handle, protocol=pickle.HIGHEST_PROTOCOL)
with open('filename.pickle', 'rb') as handle:
b = pickle.load(handle)
print(a == b)
There’s nothing about the above solution that is specific to a dict object. This same approach will will work for many Python objects, including instances of arbitrary classes and arbitrarily complex nestings of data structures. For example, replacing the second line with these lines:
import datetime
today = datetime.datetime.now()
a = [{'hello': 'world'}, 1, 2.3333, 4, True, "x",
("y", [[["z"], "y"], "x"]), {'today', today}]
will produce a result of True as well.
Some objects can’t be pickled due to their very nature. For example, it doesn’t make sense to pickle a structure containing a handle to an open file.
>>> import pickle
>>> with open("/tmp/picklefile", "wb") as f:
... pickle.dump({}, f)
...
normally it’s preferable to use the cPickle implementation
>>> import cPickle as pickle
>>> help(pickle.dump)
Help on built-in function dump in module cPickle:
dump(...)
dump(obj, file, protocol=0) -- Write an object in pickle format to the given file.
See the Pickler docstring for the meaning of optional argument proto.
import pickle
dictobj = {'Jack' : 123, 'John' : 456}
filename = "/foldername/filestore"
fileobj = open(filename, 'wb')
pickle.dump(dictobj, fileobj)
fileobj.close()
# Save a dictionary into a pickle file.
import pickle
favorite_color = {"lion": "yellow", "kitty": "red"} # create a dictionary
pickle.dump(favorite_color, open("save.p", "wb")) # save it into a file named save.p
# -------------------------------------------------------------
# Load the dictionary back from the pickle file.
import pickle
favorite_color = pickle.load(open("save.p", "rb"))
# favorite_color is now {"lion": "yellow", "kitty": "red"}
I’ve found pickling confusing (possibly because I’m thick). I found that this works, though:
myDictionaryString=str(myDictionary)
Which you can then write to a text file. I gave up trying to use pickle as I was getting errors telling me to write integers to a .dat file. I apologise for not using pickle.
In general, pickling a dict will fail unless you have only simple objects in it, like strings and integers.
Python 2.7.9 (default, Dec 11 2014, 01:21:43)
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from numpy import *
>>> type(globals())
<type 'dict'>
>>> import pickle
>>> pik = pickle.dumps(globals())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
Pickler(file, protocol).dump(obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
save(v)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 306, in save
rv = reduce(self.proto)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle module objects
>>>
Even a really simple dict will often fail. It just depends on the contents.
>>> d = {'x': lambda x:x}
>>> pik = pickle.dumps(d)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
Pickler(file, protocol).dump(obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
save(v)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 748, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle <function <lambda> at 0x102178668>: it's not found as __main__.<lambda>
However, if you use a better serializer like dill or cloudpickle, then most dictionaries can be pickled:
>>> import dill
>>> pik = dill.dumps(d)
Or if you want to save your dict to a file…
>>> with open('save.pik', 'w') as f:
... dill.dump(globals(), f)
...
The latter example is identical to any of the other good answers posted here (which aside from neglecting the picklability of the contents of the dict are good).
import pickle
your_data = {'foo': 'bar'}
# Store data (serialize)
with open('filename.pickle', 'wb') as handle:
pickle.dump(your_data, handle, protocol=pickle.HIGHEST_PROTOCOL)
# Load data (deserialize)
with open('filename.pickle', 'rb') as handle:
unserialized_data = pickle.load(handle)
print(your_data == unserialized_data)
The advantage of HIGHEST_PROTOCOL is that files get smaller. This makes unpickling sometimes much faster.
Important notice: The maximum file size of pickle is about 2GB.
Alternative way
import mpu
your_data = {'foo': 'bar'}
mpu.io.write('filename.pickle', data)
unserialized_data = mpu.io.read('filename.pickle')
Alternative Formats
- CSV: Super simple format (read & write)
- JSON: Nice for writing human-readable data; VERY commonly used (read & write)
- YAML: YAML is a superset of JSON, but easier to read (read & write, comparison of JSON and YAML)
- pickle: A Python serialization format (read & write)
- MessagePack (Python package): More compact representation (read & write)
- HDF5 (Python package): Nice for matrices (read & write)
- XML: exists too *sigh* (read & write)
For your application, the following might be important:
- Support by other programming languages
- Reading / writing performance
- Compactness (file size)
See also: Comparison of data serialization formats
In case you are rather looking for a way to make configuration files, you might want to read my short article Configuration files in Python
If you just want to store the dict in a single file, use pickle like that
import pickle
a = {'hello': 'world'}
with open('filename.pickle', 'wb') as handle:
pickle.dump(a, handle)
with open('filename.pickle', 'rb') as handle:
b = pickle.load(handle)
If you want to save and restore multiple dictionaries in multiple files for
caching and store more complex data,
use anycache.
It does all the other stuff you need around pickle
from anycache import anycache
@anycache(cachedir='path/to/files')
def myfunc(hello):
return {'hello', hello}
Anycache stores the different myfunc results depending on the arguments to
different files in cachedir and reloads them.
See the documentation for any further details.
Simple way to dump a Python data (e.g. dictionary) to a pickle file.
import pickle
your_dictionary = {}
pickle.dump(your_dictionary, open('pickle_file_name.p', 'wb'))
FYI, Pandas has a method to save pickles now.
I find it easier.
pd.to_pickle(object_to_save,'/temp/saved_pkl.pickle' )
If you want to handle writing or reading in one line without file opening:
import joblib
my_dict = {'hello': 'world'}
joblib.dump(my_dict, "my_dict.pickle") # write pickle file
my_dict_loaded = joblib.load("my_dict.pickle") # read pickle file