Does dictionary's clear() method delete all the item related objects from memory?
Question:
If a dictionary contains mutable objects or objects of custom classes (say a queryset, or a even a DateTime), then will calling clear() on the dictionary delete these objects from memory?
Does it behave differently than looping through the dict and deleting them?
eg.
consider
class MyClass(object):
'''Test Class.'''
my_obj_1 = MyClass()
my_obj_2 = MyClass()
my_dict = { 'foo' : my_obj_1, 'bar' : my_obj_2 }
then is
my_dict.clear()
same as
for key in my_dict.keys():
del my_dict[key]
?
Answers:
It is the same as calling del d['foo'] – it just deletes the entries, but it does not affect the keys or values themselves.
Of course, they may then become garbage collectible if there are no other references to them.
In your case, the two MyClass objects are shared. They will still be accessible via the my_obj_1 and my_obj_2.
Python documentation on dicts states that del d[key] removes d[key] from the dictionary while d.clear() removes every key, so basically their behavior is the same.
On the memory issue, in Python when you “delete” you are basically removing a reference to an object. When an object is not referenced by any variable nor other object or becomes unreachable, it becomes garbage and can be removed from memory. Python has a garbage collector that from time to time it does this job of checking which objects are garbage and releases the memory allocated for them.
If the object you are deleting from the dictionary is referenced by other variable then it is still reachable, thus it is not garbage so it won’t be deleted. I leave you here some links if you are interested in reading about garbage collection in general and python’s garbage collection in particular.
Does it behave differently than looping through the dict and deleting them?
It’s worth noting here that any custom class implementing the MutableMapping abstract base class gets clear() as a “free” mixin method.
The only methods you need to override in order to instantiate a MutableMapping subclass are:
__getitem__, __setitem__, __delitem__, __iter__, __len__
Since you can store the data in your mapping class any way you like, the only way clear() can figure out how to actually clear your data is by using one or more of those five methods. Now, you might have a guess as to which methods clear() is using, but why guess when we can experiment?
import collections
class MyMap(collections.MutableMapping):
def __init__(self, mydict):
self._top_secret_data = mydict
def __getitem__(self, key):
print 'getitem'
return self._top_secret_data[key]
def __setitem__(self, key, value):
raise Exception('where did you want that?')
def __len__(self):
raise Exception('a gentleman never tells')
def __delitem__(self, key):
print '[shredding intensifies]'
del self._top_secret_data[key]
def __iter__(self):
def keygen():
for key in self._top_secret_data:
print 'faster! faster!'
yield key
return iter(keygen())
Using the class defined above, it’s easy to see how clear() is implemented:
>>> m = MyMap({1:'a', 2:'b', 3:'c'})
>>> m.clear()
faster! faster!
getitem
[shredding intensifies]
faster! faster!
getitem
[shredding intensifies]
faster! faster!
getitem
[shredding intensifies]
>>>
In other words, the clear() mixin method is basically implemented as for key in self: del self[key].
Now, a disclaimer: Built-in types such as dict are implemented in C, so the dict.clear method may not be literally identical to for key in mydict: del mydict[key]. I would expect some optimization behind the scenes, perhaps a completely different strategy – but hopefully this example gives you some idea as to how you might expect a clear() method to work in Python.
There is in fact a very small difference between the two. clear() will free the memory of the hashset used in the dict, while removing the key will not.
a = dict.fromkeys(range(1000))
In [10]: sys.getsizeof(a)
Out[10]: 49432
In [11]: a.clear()
In [12]: sys.getsizeof(a)
Out[12]: 280
In [13]: a = dict.fromkeys(range(1000))
In [14]: for i in range(1000):
....: del a[i]
....:
In [15]: sys.getsizeof(a)
Out[15]: 49432
the del command delete the reference of the particular item in the list, the clear command clear all the key value pairs in a go so the functionality is same they both dereference and rest task of removing it from memory is done by the garbage collector
Have you even tried running the code? The following code with Python 3.7 throws exception! “RuntimeError: dictionary changed size during iteration”:
for key in my_dict.keys():
del my_dict[key]
All previous answers are good. I just wanted to add couple more points about the differences of del and clear:
- my_dict.clear(); removes all the items in the dictionary and makes it equivalent to empty dictionary. Note: you still can add items to it if you want!
- del my_dict; makes the object of my_dict to be deleted and garbage collection eligible (my_dict is not usable anymore)! So, if you try to add/access any item then you will get an exceptions.
-
Also, you have declared two variables my_obj_1 and my_obj_2; Even if you delete/clear my_dict, those two variables are holding references to MyClass objects and won’t be going away until my_obj_1 and my_obj_2 are out of scope; so, if MyClass objects hold memory (say list or something) then if your intention is to release the memory by deleting/clearing my_dict, it is not happening!
class MyClass(object):
”’Test Class.”’
my_obj_1 = MyClass()
my_obj_2 = MyClass()
If a dictionary contains mutable objects or objects of custom classes (say a queryset, or a even a DateTime), then will calling clear() on the dictionary delete these objects from memory?
Does it behave differently than looping through the dict and deleting them?
eg.
consider
class MyClass(object):
'''Test Class.'''
my_obj_1 = MyClass()
my_obj_2 = MyClass()
my_dict = { 'foo' : my_obj_1, 'bar' : my_obj_2 }
then is
my_dict.clear()
same as
for key in my_dict.keys():
del my_dict[key]
?
It is the same as calling del d['foo'] – it just deletes the entries, but it does not affect the keys or values themselves.
Of course, they may then become garbage collectible if there are no other references to them.
In your case, the two MyClass objects are shared. They will still be accessible via the my_obj_1 and my_obj_2.
Python documentation on dicts states that del d[key] removes d[key] from the dictionary while d.clear() removes every key, so basically their behavior is the same.
On the memory issue, in Python when you “delete” you are basically removing a reference to an object. When an object is not referenced by any variable nor other object or becomes unreachable, it becomes garbage and can be removed from memory. Python has a garbage collector that from time to time it does this job of checking which objects are garbage and releases the memory allocated for them.
If the object you are deleting from the dictionary is referenced by other variable then it is still reachable, thus it is not garbage so it won’t be deleted. I leave you here some links if you are interested in reading about garbage collection in general and python’s garbage collection in particular.
Does it behave differently than looping through the dict and
deleting them?
It’s worth noting here that any custom class implementing the MutableMapping abstract base class gets clear() as a “free” mixin method.
The only methods you need to override in order to instantiate a MutableMapping subclass are:
__getitem__, __setitem__, __delitem__, __iter__, __len__
Since you can store the data in your mapping class any way you like, the only way clear() can figure out how to actually clear your data is by using one or more of those five methods. Now, you might have a guess as to which methods clear() is using, but why guess when we can experiment?
import collections
class MyMap(collections.MutableMapping):
def __init__(self, mydict):
self._top_secret_data = mydict
def __getitem__(self, key):
print 'getitem'
return self._top_secret_data[key]
def __setitem__(self, key, value):
raise Exception('where did you want that?')
def __len__(self):
raise Exception('a gentleman never tells')
def __delitem__(self, key):
print '[shredding intensifies]'
del self._top_secret_data[key]
def __iter__(self):
def keygen():
for key in self._top_secret_data:
print 'faster! faster!'
yield key
return iter(keygen())
Using the class defined above, it’s easy to see how clear() is implemented:
>>> m = MyMap({1:'a', 2:'b', 3:'c'})
>>> m.clear()
faster! faster!
getitem
[shredding intensifies]
faster! faster!
getitem
[shredding intensifies]
faster! faster!
getitem
[shredding intensifies]
>>>
In other words, the clear() mixin method is basically implemented as for key in self: del self[key].
Now, a disclaimer: Built-in types such as dict are implemented in C, so the dict.clear method may not be literally identical to for key in mydict: del mydict[key]. I would expect some optimization behind the scenes, perhaps a completely different strategy – but hopefully this example gives you some idea as to how you might expect a clear() method to work in Python.
There is in fact a very small difference between the two. clear() will free the memory of the hashset used in the dict, while removing the key will not.
a = dict.fromkeys(range(1000))
In [10]: sys.getsizeof(a)
Out[10]: 49432
In [11]: a.clear()
In [12]: sys.getsizeof(a)
Out[12]: 280
In [13]: a = dict.fromkeys(range(1000))
In [14]: for i in range(1000):
....: del a[i]
....:
In [15]: sys.getsizeof(a)
Out[15]: 49432
the del command delete the reference of the particular item in the list, the clear command clear all the key value pairs in a go so the functionality is same they both dereference and rest task of removing it from memory is done by the garbage collector
Have you even tried running the code? The following code with Python 3.7 throws exception! “RuntimeError: dictionary changed size during iteration”:
for key in my_dict.keys():
del my_dict[key]
All previous answers are good. I just wanted to add couple more points about the differences of del and clear:
- my_dict.clear(); removes all the items in the dictionary and makes it equivalent to empty dictionary. Note: you still can add items to it if you want!
- del my_dict; makes the object of my_dict to be deleted and garbage collection eligible (my_dict is not usable anymore)! So, if you try to add/access any item then you will get an exceptions.
-
Also, you have declared two variables my_obj_1 and my_obj_2; Even if you delete/clear my_dict, those two variables are holding references to MyClass objects and won’t be going away until my_obj_1 and my_obj_2 are out of scope; so, if MyClass objects hold memory (say list or something) then if your intention is to release the memory by deleting/clearing my_dict, it is not happening!
class MyClass(object):
”’Test Class.”’my_obj_1 = MyClass()
my_obj_2 = MyClass()