Nested defaultdict of defaultdict
Question:
Is there a way to make a defaultdict also be the default for the defaultdict? (i.e. infinite-level recursive defaultdict?)
I want to be able to do:
x = defaultdict(...stuff...)
x[0][1][0]
{}
So, I can do x = defaultdict(defaultdict)
, but that’s only a second level:
x[0]
{}
x[0][0]
KeyError: 0
There are recipes that can do this. But can it be done simply just using the normal defaultdict arguments?
Note this is asking how to do an infinite-level recursive defaultdict, so it’s distinct to Python: defaultdict of defaultdict?, which was how to do a two-level defaultdict.
I’ll probably just end up using the bunch pattern, but when I realized I didn’t know how to do this, it got me interested.
Answers:
For an arbitrary number of levels:
def rec_dd():
return defaultdict(rec_dd)
>>> x = rec_dd()
>>> x['a']['b']['c']['d']
defaultdict(<function rec_dd at 0x7f0dcef81500>, {})
>>> print json.dumps(x)
{"a": {"b": {"c": {"d": {}}}}}
Of course you could also do this with a lambda, but I find lambdas to be less readable. In any case it would look like this:
rec_dd = lambda: defaultdict(rec_dd)
There is a nifty trick for doing that:
tree = lambda: defaultdict(tree)
Then you can create your x
with x = tree()
.
Similar to BrenBarn’s solution, but doesn’t contain the name of the variable tree
twice, so it works even after changes to the variable dictionary:
tree = (lambda f: f(f))(lambda a: (lambda: defaultdict(a(a))))
Then you can create each new x
with x = tree()
.
For the def
version, we can use function closure scope to protect the data structure from the flaw where existing instances stop working if the tree
name is rebound. It looks like this:
from collections import defaultdict
def tree():
def the_tree():
return defaultdict(the_tree)
return the_tree()
The other answers here tell you how to create a defaultdict
which contains “infinitely many” defaultdict
, but they fail to address what I think may have been your initial need which was to simply have a two-depth defaultdict.
You may have been looking for:
defaultdict(lambda: defaultdict(dict))
The reasons why you might prefer this construct are:
- It is more explicit than the recursive solution, and therefore likely more understandable to the reader.
- This enables the “leaf” of the
defaultdict
to be something other than a dictionary, e.g.,: defaultdict(lambda: defaultdict(list))
or defaultdict(lambda: defaultdict(set))
I would also propose more OOP-styled implementation, which supports infinite nesting as well as properly formatted repr
.
class NestedDefaultDict(defaultdict):
def __init__(self, *args, **kwargs):
super(NestedDefaultDict, self).__init__(NestedDefaultDict, *args, **kwargs)
def __repr__(self):
return repr(dict(self))
Usage:
my_dict = NestedDefaultDict()
my_dict['a']['b'] = 1
my_dict['a']['c']['d'] = 2
my_dict['b']
print(my_dict) # {'a': {'b': 1, 'c': {'d': 2}}, 'b': {}}
here is a recursive function to convert a recursive default dict to a normal dict
def defdict_to_dict(defdict, finaldict):
# pass in an empty dict for finaldict
for k, v in defdict.items():
if isinstance(v, defaultdict):
# new level created and that is the new value
finaldict[k] = defdict_to_dict(v, {})
else:
finaldict[k] = v
return finaldict
defdict_to_dict(my_rec_default_dict, {})
I based this of Andrew’s answer here.
If you are looking to load data from a json or an existing dict into the nester defaultdict see this example:
def nested_defaultdict(existing=None, **kwargs):
if existing is None:
existing = {}
if not isinstance(existing, dict):
return existing
existing = {key: nested_defaultdict(val) for key, val in existing.items()}
return defaultdict(nested_defaultdict, existing, **kwargs)
https://gist.github.com/nucklehead/2d29628bb49115f3c30e78c071207775
@nucklehead’s response can be extended to handle arrays in JSON as well:
def nested_dict(existing=None, **kwargs):
if existing is None:
existing = defaultdict()
if isinstance(existing, list):
existing = [nested_dict(val) for val in existing]
if not isinstance(existing, dict):
return existing
existing = {key: nested_dict(val) for key, val in existing.items()}
return defaultdict(nested_dict, existing, **kwargs)
Here is a function for an arbitrary base defaultdict for an arbitrary depth of nesting.
(cross posting from Can't pickle defaultdict)
def wrap_defaultdict(instance, times=1):
"""Wrap an instance an arbitrary number of `times` to create nested defaultdict.
Parameters
----------
instance - list, dict, int, collections.Counter
times - the number of nested keys above `instance`; if `times=3` dd[one][two][three] = instance
Notes
-----
using `x.copy` allows pickling (loading to ipyparallel cluster or pkldump)
- thanks https://stackoverflow.com/questions/16439301/cant-pickle-defaultdict
"""
from collections import defaultdict
def _dd(x):
return defaultdict(x.copy)
dd = defaultdict(instance)
for i in range(times-1):
dd = _dd(dd)
return dd
Based on Chris W answer, however, to address the type annotation concern, you could make it a factory function that defines the detailed types. For example this is the final solution to my problem when I was researching this question:
def frequency_map_factory() -> dict[str, dict[str, int]]:
"""
Provides a recorder of: per X:str, frequency of Y:str occurrences.
"""
return defaultdict(lambda: defaultdict(int))
Here’s a solution similar to @Stanislav’s answer that works with multiprocessing and also allows for termination of the nesting:
from collections import defaultdict
from functools import partial
class NestedDD(defaultdict):
def __init__(self, n, *args, **kwargs):
self.n = n
factory = partial(build_nested_dd, n=n - 1) if n > 1 else int
super().__init__(factory, *args, **kwargs)
def __repr__(self):
return repr(dict(self))
def build_nested_dd(n):
return NestedDD(n)
Is there a way to make a defaultdict also be the default for the defaultdict? (i.e. infinite-level recursive defaultdict?)
I want to be able to do:
x = defaultdict(...stuff...)
x[0][1][0]
{}
So, I can do x = defaultdict(defaultdict)
, but that’s only a second level:
x[0]
{}
x[0][0]
KeyError: 0
There are recipes that can do this. But can it be done simply just using the normal defaultdict arguments?
Note this is asking how to do an infinite-level recursive defaultdict, so it’s distinct to Python: defaultdict of defaultdict?, which was how to do a two-level defaultdict.
I’ll probably just end up using the bunch pattern, but when I realized I didn’t know how to do this, it got me interested.
For an arbitrary number of levels:
def rec_dd():
return defaultdict(rec_dd)
>>> x = rec_dd()
>>> x['a']['b']['c']['d']
defaultdict(<function rec_dd at 0x7f0dcef81500>, {})
>>> print json.dumps(x)
{"a": {"b": {"c": {"d": {}}}}}
Of course you could also do this with a lambda, but I find lambdas to be less readable. In any case it would look like this:
rec_dd = lambda: defaultdict(rec_dd)
There is a nifty trick for doing that:
tree = lambda: defaultdict(tree)
Then you can create your x
with x = tree()
.
Similar to BrenBarn’s solution, but doesn’t contain the name of the variable tree
twice, so it works even after changes to the variable dictionary:
tree = (lambda f: f(f))(lambda a: (lambda: defaultdict(a(a))))
Then you can create each new x
with x = tree()
.
For the def
version, we can use function closure scope to protect the data structure from the flaw where existing instances stop working if the tree
name is rebound. It looks like this:
from collections import defaultdict
def tree():
def the_tree():
return defaultdict(the_tree)
return the_tree()
The other answers here tell you how to create a defaultdict
which contains “infinitely many” defaultdict
, but they fail to address what I think may have been your initial need which was to simply have a two-depth defaultdict.
You may have been looking for:
defaultdict(lambda: defaultdict(dict))
The reasons why you might prefer this construct are:
- It is more explicit than the recursive solution, and therefore likely more understandable to the reader.
- This enables the “leaf” of the
defaultdict
to be something other than a dictionary, e.g.,:defaultdict(lambda: defaultdict(list))
ordefaultdict(lambda: defaultdict(set))
I would also propose more OOP-styled implementation, which supports infinite nesting as well as properly formatted repr
.
class NestedDefaultDict(defaultdict):
def __init__(self, *args, **kwargs):
super(NestedDefaultDict, self).__init__(NestedDefaultDict, *args, **kwargs)
def __repr__(self):
return repr(dict(self))
Usage:
my_dict = NestedDefaultDict()
my_dict['a']['b'] = 1
my_dict['a']['c']['d'] = 2
my_dict['b']
print(my_dict) # {'a': {'b': 1, 'c': {'d': 2}}, 'b': {}}
here is a recursive function to convert a recursive default dict to a normal dict
def defdict_to_dict(defdict, finaldict):
# pass in an empty dict for finaldict
for k, v in defdict.items():
if isinstance(v, defaultdict):
# new level created and that is the new value
finaldict[k] = defdict_to_dict(v, {})
else:
finaldict[k] = v
return finaldict
defdict_to_dict(my_rec_default_dict, {})
I based this of Andrew’s answer here.
If you are looking to load data from a json or an existing dict into the nester defaultdict see this example:
def nested_defaultdict(existing=None, **kwargs):
if existing is None:
existing = {}
if not isinstance(existing, dict):
return existing
existing = {key: nested_defaultdict(val) for key, val in existing.items()}
return defaultdict(nested_defaultdict, existing, **kwargs)
https://gist.github.com/nucklehead/2d29628bb49115f3c30e78c071207775
@nucklehead’s response can be extended to handle arrays in JSON as well:
def nested_dict(existing=None, **kwargs):
if existing is None:
existing = defaultdict()
if isinstance(existing, list):
existing = [nested_dict(val) for val in existing]
if not isinstance(existing, dict):
return existing
existing = {key: nested_dict(val) for key, val in existing.items()}
return defaultdict(nested_dict, existing, **kwargs)
Here is a function for an arbitrary base defaultdict for an arbitrary depth of nesting.
(cross posting from Can't pickle defaultdict)
def wrap_defaultdict(instance, times=1):
"""Wrap an instance an arbitrary number of `times` to create nested defaultdict.
Parameters
----------
instance - list, dict, int, collections.Counter
times - the number of nested keys above `instance`; if `times=3` dd[one][two][three] = instance
Notes
-----
using `x.copy` allows pickling (loading to ipyparallel cluster or pkldump)
- thanks https://stackoverflow.com/questions/16439301/cant-pickle-defaultdict
"""
from collections import defaultdict
def _dd(x):
return defaultdict(x.copy)
dd = defaultdict(instance)
for i in range(times-1):
dd = _dd(dd)
return dd
Based on Chris W answer, however, to address the type annotation concern, you could make it a factory function that defines the detailed types. For example this is the final solution to my problem when I was researching this question:
def frequency_map_factory() -> dict[str, dict[str, int]]:
"""
Provides a recorder of: per X:str, frequency of Y:str occurrences.
"""
return defaultdict(lambda: defaultdict(int))
Here’s a solution similar to @Stanislav’s answer that works with multiprocessing and also allows for termination of the nesting:
from collections import defaultdict
from functools import partial
class NestedDD(defaultdict):
def __init__(self, n, *args, **kwargs):
self.n = n
factory = partial(build_nested_dd, n=n - 1) if n > 1 else int
super().__init__(factory, *args, **kwargs)
def __repr__(self):
return repr(dict(self))
def build_nested_dd(n):
return NestedDD(n)