Understanding the difference between __getattr__ and __getattribute__
Question:
I am trying to understand the difference between __getattr__ and __getattribute__, however, I am failing at it.
The answer to the Stack Overflow question Difference between __getattr__ vs __getattribute__ says:
__getattribute__ is invoked before looking at the actual attributes on
the object, and so can be tricky to
implement correctly. You can end up in
infinite recursions very easily.
I have absolutely no idea what that means.
Then it goes on to say:
You almost certainly want __getattr__.
Why?
I read that if __getattribute__ fails, __getattr__ is called. So why are there two different methods doing the same thing? If my code implements the new style classes, what should I use?
I am looking for some code examples to clear this question. I have Googled to best of my ability, but the answers that I found don’t discuss the problem thoroughly.
If there is any documentation, I am ready to read that.
Answers:
Some basics first.
With objects, you need to deal with their attributes. Ordinarily, we do instance.attribute. Sometimes we need more control (when we do not know the name of the attribute in advance).
For example, instance.attribute would become getattr(instance, attribute_name). Using this model, we can get the attribute by supplying the attribute_name as a string.
Use of __getattr__
You can also tell a class how to deal with attributes which it doesn’t explicitly manage and do that via __getattr__ method.
Python will call this method whenever you request an attribute that hasn’t already been defined, so you can define what to do with it.
A classic use case:
class A(dict):
def __getattr__(self, name):
return self[name]
a = A()
# Now a.somekey will give a['somekey']
Caveats and use of __getattribute__
If you need to catch every attribute regardless whether it exists or not, use __getattribute__ instead. The difference is that __getattr__ only gets called for attributes that don’t actually exist. If you set an attribute directly, referencing that attribute will retrieve it without calling __getattr__.
__getattribute__ is called all the times.
__getattribute__ is called whenever an attribute access occurs.
class Foo(object):
def __init__(self, a):
self.a = 1
def __getattribute__(self, attr):
try:
return self.__dict__[attr]
except KeyError:
return 'default'
f = Foo(1)
f.a
This will cause infinite recursion. The culprit here is the line return self.__dict__[attr]. Let’s pretend (It’s close enough to the truth) that all attributes are stored in self.__dict__ and available by their name. The line
f.a
attempts to access the a attribute of f. This calls f.__getattribute__('a'). __getattribute__ then tries to load self.__dict__. __dict__ is an attribute of self == f and so python calls f.__getattribute__('__dict__') which again tries to access the attribute '__dict__‘. This is infinite recursion.
If __getattr__ had been used instead then
- It never would have run because
f has an a attribute.
- If it had run, (let’s say that you asked for
f.b) then it would not have been called to find __dict__ because it’s already there and __getattr__ is invoked only if all other methods of finding the attribute have failed.
The ‘correct’ way to write the above class using __getattribute__ is
class Foo(object):
# Same __init__
def __getattribute__(self, attr):
return super().__getattribute__(attr)
super().__getattribute__(attr) binds the __getattribute__ method of the ‘nearest’ superclass (formally, the next class in the class’s Method Resolution Order, or MRO) to the current object self and then calls it and lets that do the work.
All of this trouble is avoided by using __getattr__ which lets Python do it’s normal thing until an attribute isn’t found. At that point, Python hands control over to your __getattr__ method and lets it come up with something.
It’s also worth noting that you can run into infinite recursion with __getattr__.
class Foo(object):
def __getattr__(self, attr):
return self.attr
I’ll leave that one as an exercise.
I think the other answers have done a great job of explaining the difference between __getattr__ and __getattribute__, but one thing that might not be clear is why you would want to use __getattribute__. The cool thing about __getattribute__ is that it essentially allows you to overload the dot when accessing a class. This allows you to customize how attributes are accessed at a low level. For instance, suppose I want to define a class where all methods that only take a self argument are treated as properties:
# prop.py
import inspect
class PropClass(object):
def __getattribute__(self, attr):
val = super().__getattribute__(attr)
if callable(val):
argcount = len(inspect.getargspec(val).args)
# Account for self
if argcount == 1:
return val()
else:
return val
else:
return val
And from the interactive interpreter:
>>> import prop
>>> class A(prop.PropClass):
... def f(self):
... return 1
...
>>> a = A()
>>> a.f
1
Of course this is a silly example and you probably wouldn’t ever want to do this, but it shows you the power you can get from overriding __getattribute__.
I have gone through other’s excellent explanation. However, I found a simple answer from this blog Python Magic Methods and __getattr__. All the following are from there.
Using the __getattr__ magic method, we can intercept that inexistent attribute lookup and do something so it doesn’t fail:
class Dummy(object):
def __getattr__(self, attr):
return attr.upper()
d = Dummy()
d.does_not_exist # 'DOES_NOT_EXIST'
d.what_about_this_one # 'WHAT_ABOUT_THIS_ONE'
But if the attribute does exist, __getattr__ won’t be invoked:
class Dummy(object):
def __getattr__(self, attr):
return attr.upper()
d = Dummy()
d.value = "Python"
print(d.value) # "Python"
__getattribute__ is similar to __getattr__, with the important difference that __getattribute__ will intercept EVERY attribute lookup, doesn’t matter if the attribute exists or not.
class Dummy(object):
def __getattribute__(self, attr):
return 'YOU SEE ME?'
d = Dummy()
d.value = "Python"
print(d.value) # "YOU SEE ME?"
In that example, the d object already has an attribute value. But when we try to access it, we don’t get the original expected value (“Python”); we’re just getting whatever __getattribute__ returned. It means that we’ve virtually lost the value attribute; it has become “unreachable”.
I am trying to understand the difference between __getattr__ and __getattribute__, however, I am failing at it.
The answer to the Stack Overflow question Difference between __getattr__ vs __getattribute__ says:
__getattribute__is invoked before looking at the actual attributes on
the object, and so can be tricky to
implement correctly. You can end up in
infinite recursions very easily.
I have absolutely no idea what that means.
Then it goes on to say:
You almost certainly want
__getattr__.
Why?
I read that if __getattribute__ fails, __getattr__ is called. So why are there two different methods doing the same thing? If my code implements the new style classes, what should I use?
I am looking for some code examples to clear this question. I have Googled to best of my ability, but the answers that I found don’t discuss the problem thoroughly.
If there is any documentation, I am ready to read that.
Some basics first.
With objects, you need to deal with their attributes. Ordinarily, we do instance.attribute. Sometimes we need more control (when we do not know the name of the attribute in advance).
For example, instance.attribute would become getattr(instance, attribute_name). Using this model, we can get the attribute by supplying the attribute_name as a string.
Use of __getattr__
You can also tell a class how to deal with attributes which it doesn’t explicitly manage and do that via __getattr__ method.
Python will call this method whenever you request an attribute that hasn’t already been defined, so you can define what to do with it.
A classic use case:
class A(dict):
def __getattr__(self, name):
return self[name]
a = A()
# Now a.somekey will give a['somekey']
Caveats and use of __getattribute__
If you need to catch every attribute regardless whether it exists or not, use __getattribute__ instead. The difference is that __getattr__ only gets called for attributes that don’t actually exist. If you set an attribute directly, referencing that attribute will retrieve it without calling __getattr__.
__getattribute__ is called all the times.
__getattribute__ is called whenever an attribute access occurs.
class Foo(object):
def __init__(self, a):
self.a = 1
def __getattribute__(self, attr):
try:
return self.__dict__[attr]
except KeyError:
return 'default'
f = Foo(1)
f.a
This will cause infinite recursion. The culprit here is the line return self.__dict__[attr]. Let’s pretend (It’s close enough to the truth) that all attributes are stored in self.__dict__ and available by their name. The line
f.a
attempts to access the a attribute of f. This calls f.__getattribute__('a'). __getattribute__ then tries to load self.__dict__. __dict__ is an attribute of self == f and so python calls f.__getattribute__('__dict__') which again tries to access the attribute '__dict__‘. This is infinite recursion.
If __getattr__ had been used instead then
- It never would have run because
fhas anaattribute. - If it had run, (let’s say that you asked for
f.b) then it would not have been called to find__dict__because it’s already there and__getattr__is invoked only if all other methods of finding the attribute have failed.
The ‘correct’ way to write the above class using __getattribute__ is
class Foo(object):
# Same __init__
def __getattribute__(self, attr):
return super().__getattribute__(attr)
super().__getattribute__(attr) binds the __getattribute__ method of the ‘nearest’ superclass (formally, the next class in the class’s Method Resolution Order, or MRO) to the current object self and then calls it and lets that do the work.
All of this trouble is avoided by using __getattr__ which lets Python do it’s normal thing until an attribute isn’t found. At that point, Python hands control over to your __getattr__ method and lets it come up with something.
It’s also worth noting that you can run into infinite recursion with __getattr__.
class Foo(object):
def __getattr__(self, attr):
return self.attr
I’ll leave that one as an exercise.
I think the other answers have done a great job of explaining the difference between __getattr__ and __getattribute__, but one thing that might not be clear is why you would want to use __getattribute__. The cool thing about __getattribute__ is that it essentially allows you to overload the dot when accessing a class. This allows you to customize how attributes are accessed at a low level. For instance, suppose I want to define a class where all methods that only take a self argument are treated as properties:
# prop.py
import inspect
class PropClass(object):
def __getattribute__(self, attr):
val = super().__getattribute__(attr)
if callable(val):
argcount = len(inspect.getargspec(val).args)
# Account for self
if argcount == 1:
return val()
else:
return val
else:
return val
And from the interactive interpreter:
>>> import prop
>>> class A(prop.PropClass):
... def f(self):
... return 1
...
>>> a = A()
>>> a.f
1
Of course this is a silly example and you probably wouldn’t ever want to do this, but it shows you the power you can get from overriding __getattribute__.
I have gone through other’s excellent explanation. However, I found a simple answer from this blog Python Magic Methods and __getattr__. All the following are from there.
Using the __getattr__ magic method, we can intercept that inexistent attribute lookup and do something so it doesn’t fail:
class Dummy(object):
def __getattr__(self, attr):
return attr.upper()
d = Dummy()
d.does_not_exist # 'DOES_NOT_EXIST'
d.what_about_this_one # 'WHAT_ABOUT_THIS_ONE'
But if the attribute does exist, __getattr__ won’t be invoked:
class Dummy(object):
def __getattr__(self, attr):
return attr.upper()
d = Dummy()
d.value = "Python"
print(d.value) # "Python"
__getattribute__ is similar to __getattr__, with the important difference that __getattribute__ will intercept EVERY attribute lookup, doesn’t matter if the attribute exists or not.
class Dummy(object):
def __getattribute__(self, attr):
return 'YOU SEE ME?'
d = Dummy()
d.value = "Python"
print(d.value) # "YOU SEE ME?"
In that example, the d object already has an attribute value. But when we try to access it, we don’t get the original expected value (“Python”); we’re just getting whatever __getattribute__ returned. It means that we’ve virtually lost the value attribute; it has become “unreachable”.