How to `pop` element from PyListObject?
Question:
Lets say I have a PyListObject
, and I want to append a PyObject
. I can use PyList_Append
API which is documented in the List Objects C-API. But for my use case I want to pop
an element from the PyListObject
(i.e. my_list.pop()
in python layer).
But the List Objects C-API documentation does not have any hint about the pop
operation.
So my question is, How can I pop
an element from PyListObject
using C-API.
Answers:
You will have to roll it out by yourself. Here is a possible implementation (without error checking):
PyObject *my_pop_from_list(PyListObject *lst){
//TODO: check lst isn't empty
Py_SIZE(lst) -= 1; // forget last element
return PyList_GET_ITEM(lst, PyList_GET_SIZE(lst)); // return last element
}
Py_SIZE
is just a macro to access lst->ob_size
, which we decrease while executing pop
.
Also versions without error checking, i.e. PyList_GET_ITEM
and PyList_GET_SIZE
, are used, because once it is established (see TODO-comment), that the list isn’t empty – nothing could go wrong.
The caller receives a new reference, albeit PyList_GET_ITEM
returns a borrowed one: decreasing the size of the list the way we did in the code above, makes list “forget” the reference without decreasing the reference counter.
As @MSeifert has pointed out, this version doesn’t change the size of the underlying array, the same way list.pop()
would do (if only half or less of the underlying array is used after pop). This could be seen as “feature” of the above implementation – trading speed for memory.
No, the list.pop
method is not directly available via the C-API on PyListObject
s.
Given that list.pop
already exists and is implemented in C you could simply look up what the CPython implementation does:
static PyObject *
list_pop_impl(PyListObject *self, Py_ssize_t index)
{
PyObject *v;
int status;
if (Py_SIZE(self) == 0) {
/* Special-case most common failure cause */
PyErr_SetString(PyExc_IndexError, "pop from empty list");
return NULL;
}
if (index < 0)
index += Py_SIZE(self);
if (index < 0 || index >= Py_SIZE(self)) {
PyErr_SetString(PyExc_IndexError, "pop index out of range");
return NULL;
}
v = self->ob_item[index];
if (index == Py_SIZE(self) - 1) {
status = list_resize(self, Py_SIZE(self) - 1);
if (status >= 0)
return v; /* and v now owns the reference the list had */
else
return NULL;
}
Py_INCREF(v);
status = list_ass_slice(self, index, index+1, (PyObject *)NULL);
if (status < 0) {
Py_DECREF(v);
return NULL;
}
return v;
}
This includes a lot of functions that are not (easily) accessible for a C extension and it also handles popping from a specific index (even negative ones). Personally I wouldn’t even bother to re-implement it but just call the pop
method with PyObject_CallMethod
:
PyObject *
list_pop(PyObject *lst){
return PyObject_CallMethod(lst, "pop", "n", Py_SIZE(lst) - 1);
}
It might be a bit slower than a re-implementation but it should be “safer” – one cannot accidentally mess up invariants of the list object (for example resize conditions).
Another implementation is present in Cython
static CYTHON_INLINE PyObject* __Pyx_PyList_Pop(PyObject* L) {
/* Check that both the size is positive and no reallocation shrinking needs to be done. */
if (likely(PyList_GET_SIZE(L) > (((PyListObject*)L)->allocated >> 1))) {
Py_SIZE(L) -= 1;
return PyList_GET_ITEM(L, PyList_GET_SIZE(L));
}
return CALL_UNBOUND_METHOD(PyList_Type, "pop", L);
}
That could also be adapted for your use-case.
Lets say I have a PyListObject
, and I want to append a PyObject
. I can use PyList_Append
API which is documented in the List Objects C-API. But for my use case I want to pop
an element from the PyListObject
(i.e. my_list.pop()
in python layer).
But the List Objects C-API documentation does not have any hint about the pop
operation.
So my question is, How can I pop
an element from PyListObject
using C-API.
You will have to roll it out by yourself. Here is a possible implementation (without error checking):
PyObject *my_pop_from_list(PyListObject *lst){
//TODO: check lst isn't empty
Py_SIZE(lst) -= 1; // forget last element
return PyList_GET_ITEM(lst, PyList_GET_SIZE(lst)); // return last element
}
Py_SIZE
is just a macro to access lst->ob_size
, which we decrease while executing pop
.
Also versions without error checking, i.e. PyList_GET_ITEM
and PyList_GET_SIZE
, are used, because once it is established (see TODO-comment), that the list isn’t empty – nothing could go wrong.
The caller receives a new reference, albeit PyList_GET_ITEM
returns a borrowed one: decreasing the size of the list the way we did in the code above, makes list “forget” the reference without decreasing the reference counter.
As @MSeifert has pointed out, this version doesn’t change the size of the underlying array, the same way list.pop()
would do (if only half or less of the underlying array is used after pop). This could be seen as “feature” of the above implementation – trading speed for memory.
No, the list.pop
method is not directly available via the C-API on PyListObject
s.
Given that list.pop
already exists and is implemented in C you could simply look up what the CPython implementation does:
static PyObject *
list_pop_impl(PyListObject *self, Py_ssize_t index)
{
PyObject *v;
int status;
if (Py_SIZE(self) == 0) {
/* Special-case most common failure cause */
PyErr_SetString(PyExc_IndexError, "pop from empty list");
return NULL;
}
if (index < 0)
index += Py_SIZE(self);
if (index < 0 || index >= Py_SIZE(self)) {
PyErr_SetString(PyExc_IndexError, "pop index out of range");
return NULL;
}
v = self->ob_item[index];
if (index == Py_SIZE(self) - 1) {
status = list_resize(self, Py_SIZE(self) - 1);
if (status >= 0)
return v; /* and v now owns the reference the list had */
else
return NULL;
}
Py_INCREF(v);
status = list_ass_slice(self, index, index+1, (PyObject *)NULL);
if (status < 0) {
Py_DECREF(v);
return NULL;
}
return v;
}
This includes a lot of functions that are not (easily) accessible for a C extension and it also handles popping from a specific index (even negative ones). Personally I wouldn’t even bother to re-implement it but just call the pop
method with PyObject_CallMethod
:
PyObject *
list_pop(PyObject *lst){
return PyObject_CallMethod(lst, "pop", "n", Py_SIZE(lst) - 1);
}
It might be a bit slower than a re-implementation but it should be “safer” – one cannot accidentally mess up invariants of the list object (for example resize conditions).
Another implementation is present in Cython
static CYTHON_INLINE PyObject* __Pyx_PyList_Pop(PyObject* L) {
/* Check that both the size is positive and no reallocation shrinking needs to be done. */
if (likely(PyList_GET_SIZE(L) > (((PyListObject*)L)->allocated >> 1))) {
Py_SIZE(L) -= 1;
return PyList_GET_ITEM(L, PyList_GET_SIZE(L));
}
return CALL_UNBOUND_METHOD(PyList_Type, "pop", L);
}
That could also be adapted for your use-case.