Reference counting using PyDict_SetItemString
Question:
I’m wondering how memory management/reference counting works when a new value is set into an existing field within a PyDict (within a C extension).
For instance, assume a dictionary is created and populated in the following way:
myPyDict = PyDict_New();
tempPyObj = PyString_FromString("Original Value");
PyDict_SetItemString(myPyDict,"fieldname",tempPyObj);
Py_DECREF(tempPyObj);
From a memory and reference counting perspective, what happens when there is a subsequent
tempPyObj = PyString_FromString("New Value");
PyDict_SetItemString(myPyDict,"fieldname",tempPyObj);
Py_DECREF(tempPyObj);
Is the reference count for the original value automatically decremented (and the memory automatically released)?
The doc for PyList_SetItem specifically mentions what happens for lists: This function “steals” a reference to item and discards a reference to an item already in the list at the affected position.
But neither PyDic_SetItem nor PyDict_SetItemString say how the replacement is handled for dictionaries.
Answers:
The ref count of the old value is decremented automatically by logic of PyList_SetItem function.
You shouldn’t be decrementing the old value by yourself.
If you want to know details, take a look at CPython source code, specifically at the Objects/dictobject.c file.
1) The PyDict_SetItemString()
https://github.com/python/cpython/blob/c8e7c5a/Objects/dictobject.c#L3250-L3262
It it’s a thin wrapper around PyDict_SetItem()
2) The PyDict_SetItem()
https://github.com/python/cpython/blob/c8e7c5a/Objects/dictobject.c#L1537-L1565
Is still quite simple and it’s obvious that all the heavy-lifting related to inserting/replacing values in a dict is actually done by insertdict().
3) The insertdict()
https://github.com/python/cpython/blob/c8e7c5a/Objects/dictobject.c#L1097-L1186
is the function which gives us the answer. In this function you can find the crucial code and especially the call:
Py_XDECREF(old_value); /* which **CAN** re-enter (see issue #22653) */
which decrements the ref count of old value.
Apologies for posting in an old question, but the accepted answer seems to lead people to the wrong conclusion as seen in the comments (without actually explicitly stating whether or not the reference is stolen or not). This confused me because I’ve seen other posts saying it doesn’t, and as mentioned the docs don’t say it steals a reference either. In fact people working in the code base and the docs also explicitly state it does not steal a reference: https://bugs.python.org/issue39153
In response, I decided to look at the code linked in the accepted answer for insertdict() and while it does in fact decrement a reference to the old value, as stated in the accepted answer, it also increments a reference to the new value. Since the old value was presumably inserted at one point as well we can presume it also had its reference incremented prior to it being decremented.
This means that the reference is not stolen, and you should decrement your reference count to anything passed to the PyDict_Set functions as usual (unless otherwise noted in the docs).
Again apologies for responding to an old question, but this still comes up near the top when searching for information on references and PyDict_Set functions for older versions of Python, and I wanted to save other folks the hassle I went through.
I’m wondering how memory management/reference counting works when a new value is set into an existing field within a PyDict (within a C extension).
For instance, assume a dictionary is created and populated in the following way:
myPyDict = PyDict_New();
tempPyObj = PyString_FromString("Original Value");
PyDict_SetItemString(myPyDict,"fieldname",tempPyObj);
Py_DECREF(tempPyObj);
From a memory and reference counting perspective, what happens when there is a subsequent
tempPyObj = PyString_FromString("New Value");
PyDict_SetItemString(myPyDict,"fieldname",tempPyObj);
Py_DECREF(tempPyObj);
Is the reference count for the original value automatically decremented (and the memory automatically released)?
The doc for PyList_SetItem specifically mentions what happens for lists: This function “steals” a reference to item and discards a reference to an item already in the list at the affected position.
But neither PyDic_SetItem nor PyDict_SetItemString say how the replacement is handled for dictionaries.
The ref count of the old value is decremented automatically by logic of PyList_SetItem function.
You shouldn’t be decrementing the old value by yourself.
If you want to know details, take a look at CPython source code, specifically at the Objects/dictobject.c file.
1) The PyDict_SetItemString()
https://github.com/python/cpython/blob/c8e7c5a/Objects/dictobject.c#L3250-L3262
It it’s a thin wrapper around PyDict_SetItem()
2) The PyDict_SetItem()
https://github.com/python/cpython/blob/c8e7c5a/Objects/dictobject.c#L1537-L1565
Is still quite simple and it’s obvious that all the heavy-lifting related to inserting/replacing values in a dict is actually done by insertdict().
3) The insertdict()
https://github.com/python/cpython/blob/c8e7c5a/Objects/dictobject.c#L1097-L1186
is the function which gives us the answer. In this function you can find the crucial code and especially the call:
Py_XDECREF(old_value); /* which **CAN** re-enter (see issue #22653) */
which decrements the ref count of old value.
Apologies for posting in an old question, but the accepted answer seems to lead people to the wrong conclusion as seen in the comments (without actually explicitly stating whether or not the reference is stolen or not). This confused me because I’ve seen other posts saying it doesn’t, and as mentioned the docs don’t say it steals a reference either. In fact people working in the code base and the docs also explicitly state it does not steal a reference: https://bugs.python.org/issue39153
In response, I decided to look at the code linked in the accepted answer for insertdict() and while it does in fact decrement a reference to the old value, as stated in the accepted answer, it also increments a reference to the new value. Since the old value was presumably inserted at one point as well we can presume it also had its reference incremented prior to it being decremented.
This means that the reference is not stolen, and you should decrement your reference count to anything passed to the PyDict_Set functions as usual (unless otherwise noted in the docs).
Again apologies for responding to an old question, but this still comes up near the top when searching for information on references and PyDict_Set functions for older versions of Python, and I wanted to save other folks the hassle I went through.