Subclass dict: UserDict, dict or ABC?
Question:
What’s the difference between UserDict, dict and ABC and which one is recommended? The docs seem to deprecate UserDict?
Also it seems UserDict’s update() would use my setitem method whereas dict doesn’t? Which methods are really essential to override given I want custom setitem and getitem function?
With ABCs I’d have to implement absolutely all methods since it provides no default implementation?
I want to make a dict that does two things:
intern() all keys and values- store some of the values in an SQLite database
So which of UserDict, dict and ABC would best allow me to do this?
Answers:
Don’t use the UserDict class — you don’t need it. As the docs say, you can just subclass dict directly.
However, you still want the UserDict module, for DictMixin:
Note: DictMixin, while not officially deprecated, has been removed in Python 3, and it’s recommended in the docs that you use collections.MutableMapping. This, however, has a drawback — you need to implement more of the dictionary interface – __delitem__, __getitem__, __iter__, __len__, and __setitem__. With DictMixin, you can just implement the ones you want to change, and the rest use a default implementation.
from UserDict import DictMixin
class MyDict(DictMixin, dict):
def __setitem__(self, key, value):
print key, value # just an example
# use intern(key) or whatever here
dict.__setitem__(self, key, value) # or
# super(MyDict, self).__setitem__(key, value)
m = MyDict()
m['a'] = 'b'
# a b
m.update({'a': 'c'})
# a c
It will automatically make update use your __setitem__ as you want.
If you want a custom collection that actually holds the data, subclass dict. This is especially useful if you want to extend the interface (e.g., add methods).
None of the built-in methods will call your custom __getitem__ / __setitem__, though. If you need total control over these, create a custom class that implements the collections.MutableMapping abstract base class instead.
The ABC does not provide a means to store the actual data, only an interface with default implementations for some methods. These default implementations will, however, call your custom __getitem__ and __setitem__. You will have to use an internal dict to hold the data, and implement all abstract methods: __len__, __iter__, __getitem__, __setitem__, and __delitem__.
The class UserDict from the collections module (in Python 2, the module is called UserDict as well) is a wrapper around an internal dict, implementing the MutableMapping ABC. If you want to customize the behavior of a dict, this implementation could be a starting point.
In summary:
- MutableMapping defines the interface. Subclass this to create something that acts like a
dict. It’s totally up to you if and how you store the data.
- UserDict is an implementation of
MutableMapping using an internal “real” dict as storage. If you want a dict-like storage collection but override some methods exposed by dict, this might be a good starting point for you. But make sure to read the code to know how the basic methods are implemented, so that you are consistent when overriding a method.
- dict is “the real thing”. Subclass this if you want to extend the interface. Overriding methods to do custom things might be dangerous, as there are usually multiple ways of accessing the data, and you could end up with an inconsistent API.
Based on Secrets Recipes of the Python Ninja book
The only special thing the UserDict has beyond the normal dictionary operations is a single attribute:
data: A real dictionary to hold the contents of the UserDict class
To get to the items in the dictionary, you have to either iterate over them or call items(). While the UserDict instance supports the same methods, the view returned by items() is noticeably different:
>>> from collections import UserDict
>>> a = UserDict(a=1)
>>> d = dict(d=3) # regular dictionary for comparison
>>> for k in d:
... print(k, d[k])
...
d 3
>>> d.items()
dict_items([('d', 3)])
>>> for k in a:
... print(k, a[k])
...
a 1
>>> a.items()
ItemsView({'a': 1})
Notice that the dictionary object returns a tuple of key/values. The UserDict returns an actual dictionary object. Depending on what you are doing, this difference can be important, as is the ability to use the data attribute to access the dictionary.
I found an example of difference between dict and userdict from here: https://dev.to/0xbf/customize-your-own-dictionary-python-tips-5b47
If you override __delitem__ from dict, this will only be applied to del method, but not pop.
the reason why this happens is because Python’s built-in dict has some inline optimizations which leads pop not calling delitem.
This is quite not intuitive.
However, when you override userdict’s __delitem__, both del and pop will be affected.
UserDict is often the simplest option when you need a custom dict.
It’s tricky to overwrite dict correctly, while UserDict makes it easy. There was some discussion to remove it from Python3, but I believe it was kept for this reason. Example:
class MyDict(dict):
def __setitem__(self, key, value):
super().__setitem__(key, value * 10)
d = MyDict(a=1, b=2) # Oups MyDict.__setitem__ not called
d.update(c=3) # Oups MyDict.__setitem__ not called
d['d'] = 4 # Good!
print(d) # {'a': 1, 'b': 2, 'c': 3, 'd': 40}
UserDict inherit collections.abc.MutableMapping, so don’t have those drawback:
class MyDict(collections.UserDict):
def __setitem__(self, key, value):
super().__setitem__(key, value * 10)
d = MyDict(a=1, b=2) # Good: MyDict.__setitem__ correctly called
d.update(c=3) # Good: MyDict.__setitem__ correctly called
d['d'] = 4 # Good
print(d) # {'a': 10, 'b': 20, 'c': 30, 'd': 40}
Directly subclassing collections.abc.MutableMapping require in addition to overwrite __len__, __iter__ ,… while subclassing UserDict is much easier.
One caveat though is that UserDict is a MutableMapping, not a dict:
assert not isinstance(collections.UserDict(), dict)
assert isinstance(collections.UserDict(), collections.abc.MutableMapping)
What’s the difference between UserDict, dict and ABC and which one is recommended? The docs seem to deprecate UserDict?
Also it seems UserDict’s update() would use my setitem method whereas dict doesn’t? Which methods are really essential to override given I want custom setitem and getitem function?
With ABCs I’d have to implement absolutely all methods since it provides no default implementation?
I want to make a dict that does two things:
intern()all keys and values- store some of the values in an SQLite database
So which of UserDict, dict and ABC would best allow me to do this?
Don’t use the UserDict class — you don’t need it. As the docs say, you can just subclass dict directly.
However, you still want the UserDict module, for DictMixin:
Note: DictMixin, while not officially deprecated, has been removed in Python 3, and it’s recommended in the docs that you use collections.MutableMapping. This, however, has a drawback — you need to implement more of the dictionary interface – __delitem__, __getitem__, __iter__, __len__, and __setitem__. With DictMixin, you can just implement the ones you want to change, and the rest use a default implementation.
from UserDict import DictMixin
class MyDict(DictMixin, dict):
def __setitem__(self, key, value):
print key, value # just an example
# use intern(key) or whatever here
dict.__setitem__(self, key, value) # or
# super(MyDict, self).__setitem__(key, value)
m = MyDict()
m['a'] = 'b'
# a b
m.update({'a': 'c'})
# a c
It will automatically make update use your __setitem__ as you want.
If you want a custom collection that actually holds the data, subclass dict. This is especially useful if you want to extend the interface (e.g., add methods).
None of the built-in methods will call your custom __getitem__ / __setitem__, though. If you need total control over these, create a custom class that implements the collections.MutableMapping abstract base class instead.
The ABC does not provide a means to store the actual data, only an interface with default implementations for some methods. These default implementations will, however, call your custom __getitem__ and __setitem__. You will have to use an internal dict to hold the data, and implement all abstract methods: __len__, __iter__, __getitem__, __setitem__, and __delitem__.
The class UserDict from the collections module (in Python 2, the module is called UserDict as well) is a wrapper around an internal dict, implementing the MutableMapping ABC. If you want to customize the behavior of a dict, this implementation could be a starting point.
In summary:
- MutableMapping defines the interface. Subclass this to create something that acts like a
dict. It’s totally up to you if and how you store the data. - UserDict is an implementation of
MutableMappingusing an internal “real”dictas storage. If you want a dict-like storage collection but override some methods exposed bydict, this might be a good starting point for you. But make sure to read the code to know how the basic methods are implemented, so that you are consistent when overriding a method. - dict is “the real thing”. Subclass this if you want to extend the interface. Overriding methods to do custom things might be dangerous, as there are usually multiple ways of accessing the data, and you could end up with an inconsistent API.
Based on Secrets Recipes of the Python Ninja book
The only special thing the UserDict has beyond the normal dictionary operations is a single attribute:
data: A real dictionary to hold the contents of the UserDict class
To get to the items in the dictionary, you have to either iterate over them or call items(). While the UserDict instance supports the same methods, the view returned by items() is noticeably different:
>>> from collections import UserDict
>>> a = UserDict(a=1)
>>> d = dict(d=3) # regular dictionary for comparison
>>> for k in d:
... print(k, d[k])
...
d 3
>>> d.items()
dict_items([('d', 3)])
>>> for k in a:
... print(k, a[k])
...
a 1
>>> a.items()
ItemsView({'a': 1})
Notice that the dictionary object returns a tuple of key/values. The UserDict returns an actual dictionary object. Depending on what you are doing, this difference can be important, as is the ability to use the data attribute to access the dictionary.
I found an example of difference between dict and userdict from here: https://dev.to/0xbf/customize-your-own-dictionary-python-tips-5b47
If you override __delitem__ from dict, this will only be applied to del method, but not pop.
the reason why this happens is because Python’s built-in dict has some inline optimizations which leads pop not calling delitem.
This is quite not intuitive.
However, when you override userdict’s __delitem__, both del and pop will be affected.
UserDict is often the simplest option when you need a custom dict.
It’s tricky to overwrite dict correctly, while UserDict makes it easy. There was some discussion to remove it from Python3, but I believe it was kept for this reason. Example:
class MyDict(dict):
def __setitem__(self, key, value):
super().__setitem__(key, value * 10)
d = MyDict(a=1, b=2) # Oups MyDict.__setitem__ not called
d.update(c=3) # Oups MyDict.__setitem__ not called
d['d'] = 4 # Good!
print(d) # {'a': 1, 'b': 2, 'c': 3, 'd': 40}
UserDict inherit collections.abc.MutableMapping, so don’t have those drawback:
class MyDict(collections.UserDict):
def __setitem__(self, key, value):
super().__setitem__(key, value * 10)
d = MyDict(a=1, b=2) # Good: MyDict.__setitem__ correctly called
d.update(c=3) # Good: MyDict.__setitem__ correctly called
d['d'] = 4 # Good
print(d) # {'a': 10, 'b': 20, 'c': 30, 'd': 40}
Directly subclassing collections.abc.MutableMapping require in addition to overwrite __len__, __iter__ ,… while subclassing UserDict is much easier.
One caveat though is that UserDict is a MutableMapping, not a dict:
assert not isinstance(collections.UserDict(), dict)
assert isinstance(collections.UserDict(), collections.abc.MutableMapping)