Can't pickle defaultdict
Question:
I have a defaultdict that looks like this:
dict1 = defaultdict(lambda: defaultdict(int))
The problem is, I can’t pickle it using cPickle. One of the solution that I found here is to use module-level function instead of a lambda. My question is, what is module-level function? How can I use the dictionary with cPickle?
Answers:
Pickle wants to store all the instance attributes, and defaultdict instances store a reference to the default callable. Pickle recurses over each instance attribute.
Pickle cannot handle lambdas; pickle only ever handles data, not code, and lambdas contain code. Functions can be pickled, but just like class definitions only if the function can be imported. A function defined at the module level can be imported. Pickle just stores a string in that case, the full ‘path’ of the function to be imported and referenced when unpickling again.
You can however use partial to accomplish this:
>>> from collections import defaultdict
>>> from functools import partial
>>> pickle.loads(pickle.dumps(defaultdict(partial(defaultdict, int))))
defaultdict(<functools.partial object at 0x94dd16c>, {})
In addition to Martijn’s explanation:
A module-level function is a function which is defined at module level, that means it is not an instance method of a class, it’s not nested within another function, and it is a “real” function with a name, not a lambda function.
So, to pickle your defaultdict, create it with module-level function instead of a lambda function:
def dd():
return defaultdict(int)
dict1 = defaultdict(dd) # dd is a module-level function
than you can pickle it
tmp = pickle.dumps(dict1) # no exception
new = pickle.loads(tmp)
I’m currently doing something similar to the question poser, however, I’m using a subclass of defaultdict which has a member function that is used as the default_factory. In order to have my code work properly (I required the function to be defined at runtime), I simply added some code to prepare the object for pickling.
Instead of:
...
pickle.dump(dict, file)
...
I use this:
....
factory = dict.default_factory
dict.default_factory = None
pickle.dump(dict, file)
dict.default_factory = factory
...
This isn’t the exact code I used as my tree is an object which creates instances of the same the tree’s type as indexes are requested (so I use a recursive member function to do the pre/post pickle operations), but this pattern also answers the question.
To do this, just write the code you wanted to write. I’d use dill, which can serialize lambdas and defaultdicts. Dill can serialize almost anything in python.
>>> import dill
>>> from collections import defaultdict
>>>
>>> dict1 = defaultdict(lambda: defaultdict(int))
>>> pdict1 = dill.dumps(dict1)
>>> _dict1 = dill.loads(pdict1)
>>> _dict1
defaultdict(<function <lambda> at 0x10b31b398>, {})
If you don’t care about preserving the defaultdict type, convert it:
fname = "file.pkl"
for value in nested_default_dict:
nested_default_dict[value] = dict(nested_default_dict[value])
my_dict = dict(nested_default_dict)
with open(fname, "wb") as f:
pickle.dump(my_dict, f) # Now this will work
I think this is a great alternative since when you are pickling, the object is probably in it’s final form… AND, if really do need the defaultdict type again, you can simply convert is back after you unpickle:
for value in my_dict:
my_dict[value] = defaultdict(type, my_dict[value])
nested_default_dict = defaultdict(type, my_dict)
dict1 = defaultdict(lambda: defaultdict(int))
cPickle.dump(dict(dict1), file_handle)
worked for me
Implementing the anonymous lambda function by a normal function worked for me. As pointed out by Mike, Pickle cannot handle lambdas; pickle only handles data. Hence, converting the defaultdict method from:
dict_ = defaultdict(lambda: default_value)
to:
def default_():
return default_value
and then creating the default dict as follows worked for me:
dict_ = defaultdict(default_)
Solution that still works as a one-liner for this case, and is actually more efficient than the lambda (or an equivalent def-ed) function to boot:
dict1 = defaultdict(defaultdict(int).copy)
That just makes a template defaultdict(int), and binds its copy method as the default factory for the outer defaultdict. Everything in there is picklable, and on CPython (where defaultdict is a built-in type implemented in C) it’s more efficient than invoking any user-defined function to do the same job. No need for extra imports, wrapping, etc.
Here is a function for an arbitrary base defaultdict for an arbitrary depth of nesting.
def wrap_defaultdict(instance, times):
"""Wrap an instance an arbitrary number of `times` to create nested defaultdict.
Parameters
----------
instance - e.g., list, dict, int, collections.Counter
times - the number of nested keys above `instance`; if `times=3` dd[one][two][three] = instance
Notes
-----
using `x.copy` allows pickling (loading to ipyparallel cluster or pkldump)
- thanks https://stackoverflow.com/questions/16439301/cant-pickle-defaultdict
"""
from collections import defaultdict
def _dd(x):
return defaultdict(x.copy)
dd = defaultdict(instance)
for i in range(times-1):
dd = _dd(dd)
return dd
I have a defaultdict that looks like this:
dict1 = defaultdict(lambda: defaultdict(int))
The problem is, I can’t pickle it using cPickle. One of the solution that I found here is to use module-level function instead of a lambda. My question is, what is module-level function? How can I use the dictionary with cPickle?
Pickle wants to store all the instance attributes, and defaultdict instances store a reference to the default callable. Pickle recurses over each instance attribute.
Pickle cannot handle lambdas; pickle only ever handles data, not code, and lambdas contain code. Functions can be pickled, but just like class definitions only if the function can be imported. A function defined at the module level can be imported. Pickle just stores a string in that case, the full ‘path’ of the function to be imported and referenced when unpickling again.
You can however use partial to accomplish this:
>>> from collections import defaultdict
>>> from functools import partial
>>> pickle.loads(pickle.dumps(defaultdict(partial(defaultdict, int))))
defaultdict(<functools.partial object at 0x94dd16c>, {})
In addition to Martijn’s explanation:
A module-level function is a function which is defined at module level, that means it is not an instance method of a class, it’s not nested within another function, and it is a “real” function with a name, not a lambda function.
So, to pickle your defaultdict, create it with module-level function instead of a lambda function:
def dd():
return defaultdict(int)
dict1 = defaultdict(dd) # dd is a module-level function
than you can pickle it
tmp = pickle.dumps(dict1) # no exception
new = pickle.loads(tmp)
I’m currently doing something similar to the question poser, however, I’m using a subclass of defaultdict which has a member function that is used as the default_factory. In order to have my code work properly (I required the function to be defined at runtime), I simply added some code to prepare the object for pickling.
Instead of:
...
pickle.dump(dict, file)
...
I use this:
....
factory = dict.default_factory
dict.default_factory = None
pickle.dump(dict, file)
dict.default_factory = factory
...
This isn’t the exact code I used as my tree is an object which creates instances of the same the tree’s type as indexes are requested (so I use a recursive member function to do the pre/post pickle operations), but this pattern also answers the question.
To do this, just write the code you wanted to write. I’d use dill, which can serialize lambdas and defaultdicts. Dill can serialize almost anything in python.
>>> import dill
>>> from collections import defaultdict
>>>
>>> dict1 = defaultdict(lambda: defaultdict(int))
>>> pdict1 = dill.dumps(dict1)
>>> _dict1 = dill.loads(pdict1)
>>> _dict1
defaultdict(<function <lambda> at 0x10b31b398>, {})
If you don’t care about preserving the defaultdict type, convert it:
fname = "file.pkl"
for value in nested_default_dict:
nested_default_dict[value] = dict(nested_default_dict[value])
my_dict = dict(nested_default_dict)
with open(fname, "wb") as f:
pickle.dump(my_dict, f) # Now this will work
I think this is a great alternative since when you are pickling, the object is probably in it’s final form… AND, if really do need the defaultdict type again, you can simply convert is back after you unpickle:
for value in my_dict:
my_dict[value] = defaultdict(type, my_dict[value])
nested_default_dict = defaultdict(type, my_dict)
dict1 = defaultdict(lambda: defaultdict(int))
cPickle.dump(dict(dict1), file_handle)
worked for me
Implementing the anonymous lambda function by a normal function worked for me. As pointed out by Mike, Pickle cannot handle lambdas; pickle only handles data. Hence, converting the defaultdict method from:
dict_ = defaultdict(lambda: default_value)
to:
def default_():
return default_value
and then creating the default dict as follows worked for me:
dict_ = defaultdict(default_)
Solution that still works as a one-liner for this case, and is actually more efficient than the lambda (or an equivalent def-ed) function to boot:
dict1 = defaultdict(defaultdict(int).copy)
That just makes a template defaultdict(int), and binds its copy method as the default factory for the outer defaultdict. Everything in there is picklable, and on CPython (where defaultdict is a built-in type implemented in C) it’s more efficient than invoking any user-defined function to do the same job. No need for extra imports, wrapping, etc.
Here is a function for an arbitrary base defaultdict for an arbitrary depth of nesting.
def wrap_defaultdict(instance, times):
"""Wrap an instance an arbitrary number of `times` to create nested defaultdict.
Parameters
----------
instance - e.g., list, dict, int, collections.Counter
times - the number of nested keys above `instance`; if `times=3` dd[one][two][three] = instance
Notes
-----
using `x.copy` allows pickling (loading to ipyparallel cluster or pkldump)
- thanks https://stackoverflow.com/questions/16439301/cant-pickle-defaultdict
"""
from collections import defaultdict
def _dd(x):
return defaultdict(x.copy)
dd = defaultdict(instance)
for i in range(times-1):
dd = _dd(dd)
return dd