Descriptors as instance attributes in python
Question:
To the question:
Why can’t descriptors be instance attributes?
it has been answered that:
descriptor objects needs to live in the class, not in the instance
because that is the way that the __getattribute__ is implemented.
A simple example. Consider a descriptor:
class Prop(object):
def __get__(self, obj, objtype=None):
if obj is None:
return self
return obj._value * obj._multiplier
def __set__(self, obj, value):
if obj is None:
return self
obj._value = value
class Obj(object):
val = Prop()
def __init__(self):
self._value = 1
self._multiplier = 0
Consider the case in which each obj has multiple Prop: I would need to use unique names to identify the values and multipliers (Like here. Having a per instance descriptor object would allow to store the _multiplier (and the _value) in the descriptor itself, simplifying a few things.
To implement per instance descriptor attributes you need to either:
I am aware that similar questions have been raised before, but I have not found a real explanation:
- Why Python is designed this way?
- What is the suggested way to store information that the descriptor needs but is per instance?
Answers:
This exact question was raised on Python-list earlier this year. I’m just going to quote Ian G. Kelly’s response:
The behavior is by design. First, keeping object behavior in the
class definition simplifies the implementation and also makes instance
checks more meaningful. To borrow your Register example, if the “M”
descriptor is defined by some instances rather than by the class, then
knowing that the object “reg” is an instance of Register does not tell
me anything about whether “reg.M” is a valid attribute or an error. As
a result, I’ll need to guard virtually every access of “reg.M” with a
try-except construct just in case “reg” is the wrong kind of register.
Second, the separation of class from instance also helps you keep
object behavior separate from object data. Consider the following
class:
class ObjectHolder(object):
def __init__(self, obj):
self.obj = obj
Don’t worry about what this class might be useful for. Just know that
it’s meant to hold and provide unrestricted access to arbitrary Python
objects:
>>> holder = ObjectHolder(42)
>>> print(holder.obj) 42
>>> holder.obj = range(5)
>>> print(holder.obj) [0, 1, 2, 3, 4]
Since the class is meant to hold arbitrary objects, it’s even valid
that somebody might want to store a descriptor object there:
>>> holder.obj = property(lambda x: x.foo)
>>> print(holder.obj) <property object at 0x02415AE0>
Now suppose that Python invoked the descriptor protocol for
descriptors stored in instance attributes:
>>> holder = ObjectHolder(None)
>>> holder.obj = property(lambda x: x.foo)
>>> print(holder.obj)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'ObjectHolder' object has no attribute 'foo'
In this case, the ObjectHolder would fail to simply hold the property
object as data. The mere act of assigning the property object, a
descriptor, to an instance attribute would change the behavior of
the ObjectHolder. Instead of treating “holder.obj” as a simple data
attribute, it would start invoking the descriptor protocol on accesses
to “holder.obj” and ultimately redirect them to the non-existent and
meaningless “holder.foo” attribute, which is certainly not what the
author of the class intended.
If you want to be able to support multiple instances of a descriptor, just make that descriptor’s constructor take a name argument (prefix), and prefix the added attributes with that name. You could even create a namespace object (dictionary) within the class instance to hold all of the new property instances.
Plenty of advanced functionality only works when defined on a class rather than an instance; all of the special methods, for example. As well as making code evaluation more efficient, this makes clear the separation between instances and types which otherwise would tend to collapse (because of course all types are objects).
I’m not sure how recommended this is, but you could in the instance store a mapping from descriptor instance to attribute value:
class Prop(object):
def __get__(self, obj, objtype=None):
if obj is None:
return self
return obj._value * obj._multiplier[self]
def __set__(self, obj, value):
if obj is None:
return self
obj._value = value
class Obj(object):
val = Prop()
def __init__(self):
self._value = 1
self._multiplier = {Obj.val: 0}
This has obvious advantages over the other two suggested options:
- per-instance classes break object orientation and increase memory usage;
- overriding
__getattribute__ is inefficient (as all attribute access must go through the overridden special method) and is fragile.
As an alternative, you could use a proxy property:
class PerInstancePropertyProxy(object):
def __init__(self, prop):
self.prop = prop
def __get__(self, instance, owner):
if instance is None:
return self
return instance.__dict__[self.prop].__get__(instance, owner)
def __set__(self, instance, value):
instance.__dict__[self.prop].__set__(instance, value)
class Prop(object):
def __init__(self, value, multiplier):
self.value = value
self.multiplier = multiplier
def __get__(self, instance, owner):
if instance is None:
return self
return self.value * self.multiplier
def __set__(self, instance, value):
self.value = value
class Obj(object):
val = PerInstancePropertyProxy('val')
def __init__(self):
self.__dict__['val'] = Prop(1.0, 10.0)
def prop(self, attr_name):
return self.__dict__[attr_name]
In Python 3.6 this can be done quite easily. Maybe it’s not as intended but hey, if it works, right? 😉
Python 3.6 adds the __set_name__ method:
object.__set_name__(self, owner, name)
Called at the time the owning class owner is created. The descriptor has been assigned to name.
New in version 3.6.
Using this name to store the internal value in the instance’s dict seems to work fine.
>>> class Prop:
... def __set_name__(self, owner, name):
... self.name = name
... def __get__(self, instance, owner):
... print('get')
... return instance.__dict__.setdefault(self.name, None)
... def __set__(self, instance, value):
... print('set')
... instance.__dict__[self.name] = value
...
>>> class A:
... prop = Prop()
...
>>> a = A()
>>> a.prop = 'spam'
set
>>> a.prop
get
'spam'
Note that this is not a full descriptor implementation and of course if you decide to use it it’s at your own risk.
To the question:
Why can’t descriptors be instance attributes?
it has been answered that:
descriptor objects needs to live in the class, not in the instance
because that is the way that the __getattribute__ is implemented.
A simple example. Consider a descriptor:
class Prop(object):
def __get__(self, obj, objtype=None):
if obj is None:
return self
return obj._value * obj._multiplier
def __set__(self, obj, value):
if obj is None:
return self
obj._value = value
class Obj(object):
val = Prop()
def __init__(self):
self._value = 1
self._multiplier = 0
Consider the case in which each obj has multiple Prop: I would need to use unique names to identify the values and multipliers (Like here. Having a per instance descriptor object would allow to store the _multiplier (and the _value) in the descriptor itself, simplifying a few things.
To implement per instance descriptor attributes you need to either:
I am aware that similar questions have been raised before, but I have not found a real explanation:
- Why Python is designed this way?
- What is the suggested way to store information that the descriptor needs but is per instance?
This exact question was raised on Python-list earlier this year. I’m just going to quote Ian G. Kelly’s response:
The behavior is by design. First, keeping object behavior in the
class definition simplifies the implementation and also makes instance
checks more meaningful. To borrow your Register example, if the “M”
descriptor is defined by some instances rather than by the class, then
knowing that the object “reg” is an instance of Register does not tell
me anything about whether “reg.M” is a valid attribute or an error. As
a result, I’ll need to guard virtually every access of “reg.M” with a
try-except construct just in case “reg” is the wrong kind of register.Second, the separation of class from instance also helps you keep
object behavior separate from object data. Consider the following
class:class ObjectHolder(object): def __init__(self, obj): self.obj = objDon’t worry about what this class might be useful for. Just know that
it’s meant to hold and provide unrestricted access to arbitrary Python
objects:>>> holder = ObjectHolder(42) >>> print(holder.obj) 42 >>> holder.obj = range(5) >>> print(holder.obj) [0, 1, 2, 3, 4]Since the class is meant to hold arbitrary objects, it’s even valid
that somebody might want to store a descriptor object there:>>> holder.obj = property(lambda x: x.foo) >>> print(holder.obj) <property object at 0x02415AE0>Now suppose that Python invoked the descriptor protocol for
descriptors stored in instance attributes:>>> holder = ObjectHolder(None) >>> holder.obj = property(lambda x: x.foo) >>> print(holder.obj) Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'ObjectHolder' object has no attribute 'foo'In this case, the ObjectHolder would fail to simply hold the property
object as data. The mere act of assigning the property object, a
descriptor, to an instance attribute would change the behavior of
the ObjectHolder. Instead of treating “holder.obj” as a simple data
attribute, it would start invoking the descriptor protocol on accesses
to “holder.obj” and ultimately redirect them to the non-existent and
meaningless “holder.foo” attribute, which is certainly not what the
author of the class intended.
If you want to be able to support multiple instances of a descriptor, just make that descriptor’s constructor take a name argument (prefix), and prefix the added attributes with that name. You could even create a namespace object (dictionary) within the class instance to hold all of the new property instances.
Plenty of advanced functionality only works when defined on a class rather than an instance; all of the special methods, for example. As well as making code evaluation more efficient, this makes clear the separation between instances and types which otherwise would tend to collapse (because of course all types are objects).
I’m not sure how recommended this is, but you could in the instance store a mapping from descriptor instance to attribute value:
class Prop(object):
def __get__(self, obj, objtype=None):
if obj is None:
return self
return obj._value * obj._multiplier[self]
def __set__(self, obj, value):
if obj is None:
return self
obj._value = value
class Obj(object):
val = Prop()
def __init__(self):
self._value = 1
self._multiplier = {Obj.val: 0}
This has obvious advantages over the other two suggested options:
- per-instance classes break object orientation and increase memory usage;
- overriding
__getattribute__is inefficient (as all attribute access must go through the overridden special method) and is fragile.
As an alternative, you could use a proxy property:
class PerInstancePropertyProxy(object):
def __init__(self, prop):
self.prop = prop
def __get__(self, instance, owner):
if instance is None:
return self
return instance.__dict__[self.prop].__get__(instance, owner)
def __set__(self, instance, value):
instance.__dict__[self.prop].__set__(instance, value)
class Prop(object):
def __init__(self, value, multiplier):
self.value = value
self.multiplier = multiplier
def __get__(self, instance, owner):
if instance is None:
return self
return self.value * self.multiplier
def __set__(self, instance, value):
self.value = value
class Obj(object):
val = PerInstancePropertyProxy('val')
def __init__(self):
self.__dict__['val'] = Prop(1.0, 10.0)
def prop(self, attr_name):
return self.__dict__[attr_name]
In Python 3.6 this can be done quite easily. Maybe it’s not as intended but hey, if it works, right? 😉
Python 3.6 adds the __set_name__ method:
object.__set_name__(self, owner, name)Called at the time the owning class owner is created. The descriptor has been assigned to name.
New in version 3.6.
Using this name to store the internal value in the instance’s dict seems to work fine.
>>> class Prop:
... def __set_name__(self, owner, name):
... self.name = name
... def __get__(self, instance, owner):
... print('get')
... return instance.__dict__.setdefault(self.name, None)
... def __set__(self, instance, value):
... print('set')
... instance.__dict__[self.name] = value
...
>>> class A:
... prop = Prop()
...
>>> a = A()
>>> a.prop = 'spam'
set
>>> a.prop
get
'spam'
Note that this is not a full descriptor implementation and of course if you decide to use it it’s at your own risk.