Are Python 3.11 objects as light as slots?
Question:
After Mark Shannon’s optimisation of Python objects, is a plain object different from an object with slots?
I understand that after this optimisation in a normal use case, the objects have no dictionary. Have the new Python objects made it unnecessary to use slots at all?
Answers:
No, __slots__ still produces more compact objects.
With __slots__ for attributes, an object’s memory layout only needs one pointer per slot. With the new lazy __dict__ creation, an object needs to store a PyDictValues object and a pointer to that PyDictValues object.
The PyDictValues object contains room for a number of pointers based on the "usable size" of the shared keys object when the PyDictValues object is created, which is usually more pointers than what you would have gotten with __slots__. It also holds some extra metadata and padding, stored before those pointers: a "prefix size" representing the size of this metadata, a "used size" representing the number of values stored, and an array of bytes tracking insertion order. The padding is used to ensure this extra metadata doesn’t interfere with the alignment of the pointers. (None of this is reflected in the PyDictValues struct definition, since C struct definitions aren’t expressive enough for this – it mostly has to be handled manually.)
So, without __slots__, you’ve got an extra PyDictValues * in the object directly, usually more room allocated for attribute pointers than necessary, and a bunch of extra metadata in the PyDictValues itself, relative to using __slots__.
Plus, with __slots__, if you don’t explicitly declare a __weakref__ slot, you don’t get one, saving some memory at the cost of not being able to create weak references to your objects. With no __slots__, Python will automatically add a __weakref__ slot to your objects.
After Mark Shannon’s optimisation of Python objects, is a plain object different from an object with slots?
I understand that after this optimisation in a normal use case, the objects have no dictionary. Have the new Python objects made it unnecessary to use slots at all?
No, __slots__ still produces more compact objects.
With __slots__ for attributes, an object’s memory layout only needs one pointer per slot. With the new lazy __dict__ creation, an object needs to store a PyDictValues object and a pointer to that PyDictValues object.
The PyDictValues object contains room for a number of pointers based on the "usable size" of the shared keys object when the PyDictValues object is created, which is usually more pointers than what you would have gotten with __slots__. It also holds some extra metadata and padding, stored before those pointers: a "prefix size" representing the size of this metadata, a "used size" representing the number of values stored, and an array of bytes tracking insertion order. The padding is used to ensure this extra metadata doesn’t interfere with the alignment of the pointers. (None of this is reflected in the PyDictValues struct definition, since C struct definitions aren’t expressive enough for this – it mostly has to be handled manually.)
So, without __slots__, you’ve got an extra PyDictValues * in the object directly, usually more room allocated for attribute pointers than necessary, and a bunch of extra metadata in the PyDictValues itself, relative to using __slots__.
Plus, with __slots__, if you don’t explicitly declare a __weakref__ slot, you don’t get one, saving some memory at the cost of not being able to create weak references to your objects. With no __slots__, Python will automatically add a __weakref__ slot to your objects.