Why am I getting an error about my class defining __slots__ when trying to pickle an object?
Question:
I’m trying to pickle an object of a (new-style) class I defined. But I’m getting the following error:
>>> with open('temp/connection.pickle','w') as f:
... pickle.dump(c,f)
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/usr/lib/python2.5/pickle.py", line 1362, in dump
Pickler(file, protocol).dump(obj)
File "/usr/lib/python2.5/pickle.py", line 224, in dump
self.save(obj)
File "/usr/lib/python2.5/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python2.5/pickle.py", line 419, in save_reduce
save(state)
File "/usr/lib/python2.5/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.5/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.5/pickle.py", line 663, in _batch_setitems
save(v)
File "/usr/lib/python2.5/pickle.py", line 306, in save
rv = reduce(self.proto)
File "/usr/lib/python2.5/copy_reg.py", line 76, in _reduce_ex
raise TypeError("a class that defines __slots__ without "
TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled
I didn’t explicitly define __slots__ in my class. Did something I do implicitly define it? How do I work around this? Do I need to define __getstate__?
Update: gnibbler chose a good example. The class of the object I’m trying to pickle wraps a socket. (It occurs to me now that) sockets define __slots__ and not __getstate__ for good reason. I assume once a process ends, another process can’t unpickle and use the previous process’s socket connection. So while I’m accepting Alex Martelli‘s excellent answer, I’m going to have to pursue a different strategy than pickling to “share” the object reference.
Answers:
From PEP 307:
The __getstate__ method should return a picklable value
representing the object’s state without referencing the object
itself. If no __getstate__ method exists, a default
implementation is used that returns self.__dict__.
Perhaps an attribute of your instance is using __slots__
For example, socket has __slots__ so it can’t be pickled
You need to identify which attribute is causing the error and write your own
__getstate__ and __setstate__ to ignore that attribute
The class defining __slots__ (and not __getstate__) can be either an ancestor class of yours, or a class (or ancestor class) of an attribute or item of yours, directly or indirectly: essentially, the class of any object in the directed graph of references with your object as root, since pickling needs to save the entire graph.
A simple solution to your quandary is to use protocol -1, which means “the best protocol pickle can use”; the default is an ancient ASCII-based protocol which imposes this limitation about __slots__ vs __getstate__. Consider:
>>> class sic(object):
... __slots__ = 'a', 'b'
...
>>> import pickle
>>> pickle.dumps(sic(), -1)
'x80x02c__main__nsicnqx00)x81qx01.'
>>> pickle.dumps(sic())
Traceback (most recent call last):
[snip snip]
raise TypeError("a class that defines __slots__ without "
TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled
>>>
As you see, protocol -1 takes the __slots__ in stride, while the default protocol gives the same exception you saw.
The issues with protocol -1: it produces a binary string/file, rather than an ASCII one like the default protocol; the resulting pickled file would not be loadable by sufficiently ancient versions of Python. Advantages, besides the key one wrt __slots__, include more compact results, and better performance.
If you’re forced to use the default protocol, then you’ll need to identify exactly which class is giving you trouble and exactly why. We can discuss strategies if this is the case (but if you can possibly use the -1 protocol, that’s so much better that it’s not worth discussing;-) and simple code inspection looking for the troublesome class/object is proving too complicated (I have in mind some deepcopy-based tricks to get a usable representation of the whole graph, in case you’re wondering).
I’m trying to pickle an object of a (new-style) class I defined. But I’m getting the following error:
>>> with open('temp/connection.pickle','w') as f:
... pickle.dump(c,f)
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/usr/lib/python2.5/pickle.py", line 1362, in dump
Pickler(file, protocol).dump(obj)
File "/usr/lib/python2.5/pickle.py", line 224, in dump
self.save(obj)
File "/usr/lib/python2.5/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python2.5/pickle.py", line 419, in save_reduce
save(state)
File "/usr/lib/python2.5/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.5/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/usr/lib/python2.5/pickle.py", line 663, in _batch_setitems
save(v)
File "/usr/lib/python2.5/pickle.py", line 306, in save
rv = reduce(self.proto)
File "/usr/lib/python2.5/copy_reg.py", line 76, in _reduce_ex
raise TypeError("a class that defines __slots__ without "
TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled
I didn’t explicitly define __slots__ in my class. Did something I do implicitly define it? How do I work around this? Do I need to define __getstate__?
Update: gnibbler chose a good example. The class of the object I’m trying to pickle wraps a socket. (It occurs to me now that) sockets define __slots__ and not __getstate__ for good reason. I assume once a process ends, another process can’t unpickle and use the previous process’s socket connection. So while I’m accepting Alex Martelli‘s excellent answer, I’m going to have to pursue a different strategy than pickling to “share” the object reference.
From PEP 307:
The
__getstate__method should return a picklable value
representing the object’s state without referencing the object
itself. If no__getstate__method exists, a default
implementation is used that returnsself.__dict__.
Perhaps an attribute of your instance is using __slots__
For example, socket has __slots__ so it can’t be pickled
You need to identify which attribute is causing the error and write your own
__getstate__ and __setstate__ to ignore that attribute
The class defining __slots__ (and not __getstate__) can be either an ancestor class of yours, or a class (or ancestor class) of an attribute or item of yours, directly or indirectly: essentially, the class of any object in the directed graph of references with your object as root, since pickling needs to save the entire graph.
A simple solution to your quandary is to use protocol -1, which means “the best protocol pickle can use”; the default is an ancient ASCII-based protocol which imposes this limitation about __slots__ vs __getstate__. Consider:
>>> class sic(object):
... __slots__ = 'a', 'b'
...
>>> import pickle
>>> pickle.dumps(sic(), -1)
'x80x02c__main__nsicnqx00)x81qx01.'
>>> pickle.dumps(sic())
Traceback (most recent call last):
[snip snip]
raise TypeError("a class that defines __slots__ without "
TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled
>>>
As you see, protocol -1 takes the __slots__ in stride, while the default protocol gives the same exception you saw.
The issues with protocol -1: it produces a binary string/file, rather than an ASCII one like the default protocol; the resulting pickled file would not be loadable by sufficiently ancient versions of Python. Advantages, besides the key one wrt __slots__, include more compact results, and better performance.
If you’re forced to use the default protocol, then you’ll need to identify exactly which class is giving you trouble and exactly why. We can discuss strategies if this is the case (but if you can possibly use the -1 protocol, that’s so much better that it’s not worth discussing;-) and simple code inspection looking for the troublesome class/object is proving too complicated (I have in mind some deepcopy-based tricks to get a usable representation of the whole graph, in case you’re wondering).