python: are property fields being cached automatically?
Question:
My question is are the following two pieces of code run the same by the interpreter:
class A(object):
def __init__(self):
self.__x = None
@property
def x(self):
if not self.__x:
self.__x = ... #some complicated action
return self.__x
and the much simpler:
class A(object):
@property
def x(self):
return ... #some complicated action
I.e., is the interpreter smart enough to cache the property x?
My assumption is that x does not change – finding it is hard, but once you find it once there is no reason to find it again.
Answers:
No, the getter will be called every time you access the property.
No you need to add a memoize decorator:
class memoized(object):
"""Decorator that caches a function's return value each time it is called.
If called later with the same arguments, the cached value is returned, and
not re-evaluated.
"""
def __init__(self, func):
self.func = func
self.cache = {}
def __call__(self, *args):
try:
return self.cache[args]
except KeyError:
value = self.func(*args)
self.cache[args] = value
return value
except TypeError:
# uncachable -- for instance, passing a list as an argument.
# Better to not cache than to blow up entirely.
return self.func(*args)
def __repr__(self):
"""Return the function's docstring."""
return self.func.__doc__
def __get__(self, obj, objtype):
"""Support instance methods."""
return functools.partial(self.__call__, obj)
@memoized
def fibonacci(n):
"Return the nth fibonacci number."
if n in (0, 1):
return n
return fibonacci(n-1) + fibonacci(n-2)
print fibonacci(12)
Properties do not automatically cache their return values. The getter (and setters) are intended to be called each time the property is accessed.
However, Denis Otkidach has written a wonderful cached attribute decorator (published in the Python Cookbook, 2nd edition and also originally on ActiveState under the PSF license) for just this purpose:
class cache(object):
'''Computes attribute value and caches it in the instance.
Python Cookbook (Denis Otkidach) https://stackoverflow.com/users/168352/denis-otkidach
This decorator allows you to create a property which can be computed once and
accessed many times. Sort of like memoization.
'''
def __init__(self, method, name=None):
# record the unbound-method and the name
self.method = method
self.name = name or method.__name__
self.__doc__ = method.__doc__
def __get__(self, inst, cls):
# self: <__main__.cache object at 0xb781340c>
# inst: <__main__.Foo object at 0xb781348c>
# cls: <class '__main__.Foo'>
if inst is None:
# instance attribute accessed on class, return self
# You get here if you write `Foo.bar`
return self
# compute, cache and return the instance's attribute value
result = self.method(inst)
# setattr redefines the instance's attribute so this doesn't get called again
setattr(inst, self.name, result)
return result
Here is an example demonstrating its use:
def demo_cache():
class Foo(object):
@cache
def bar(self):
print 'Calculating self.bar'
return 42
foo=Foo()
print(foo.bar)
# Calculating self.bar
# 42
print(foo.bar)
# 42
foo.bar=1
print(foo.bar)
# 1
print(Foo.bar)
# __get__ called with inst = None
# <__main__.cache object at 0xb7709b4c>
# Deleting `foo.bar` from `foo.__dict__` re-exposes the property defined in `Foo`.
# Thus, calling `foo.bar` again recalculates the value again.
del foo.bar
print(foo.bar)
# Calculating self.bar
# 42
demo_cache()
The decorator from Denis Otkidach mentioned in @unutbu’s answer was published in O’Reilly’s Python Cookbook. Unfortunately O’Reilly doesn’t specify any license for code examples – just as informal permission to reuse the code.
If you need a cached property decorator with a liberal license, you can use Ken Seehof‘s @cached_property from ActiveState code recipes. It’s explicitly published under the MIT license.
def cached_property(f):
"""returns a cached property that is calculated by function f"""
def get(self):
try:
return self._property_cache[f]
except AttributeError:
self._property_cache = {}
x = self._property_cache[f] = f(self)
return x
except KeyError:
x = self._property_cache[f] = f(self)
return x
return property(get)
Python 3.2 onwards offers a built-in decorator that you can use to create a LRU cache:
@functools.lru_cache(maxsize=128, typed=False)
Alternatively, if you’re using Flask / Werkzeug, there’s the @cached_property decorator.
For Django, try from django.utils.functional import cached_property
Note: Adding for the sake of completeness of available options.
No, property is not cached by default. However there are several options to get that behaviour, I would like to add one more to that:
I’ve had to look it up, since I had this same question.
The functools package from the standard library will be getting a cached_property decorator as well. Unfortunately, it’s only available from Python 3.8 (as of time of this post, it’s 3.8a0). The alternative to waiting is to use a custom one, such as this one as mentioned by 0xc0de) or Django’s, for now, then switch later:
from django.utils.functional import cached_property
# from functools import cached_property # Only 3.8+ :(
To anyone who might be reading this in 2020, this functionality is now available in the functools module as part of the standard library as of Python 3.8.
https://docs.python.org/dev/library/functools.html#functools.cached_property
Important to note, classes that define their own __dict__ (or do not define one at all) or use __slots__ might not work as expected. For example, NamedTuple and metaclasses.
My question is are the following two pieces of code run the same by the interpreter:
class A(object):
def __init__(self):
self.__x = None
@property
def x(self):
if not self.__x:
self.__x = ... #some complicated action
return self.__x
and the much simpler:
class A(object):
@property
def x(self):
return ... #some complicated action
I.e., is the interpreter smart enough to cache the property x?
My assumption is that x does not change – finding it is hard, but once you find it once there is no reason to find it again.
No, the getter will be called every time you access the property.
No you need to add a memoize decorator:
class memoized(object):
"""Decorator that caches a function's return value each time it is called.
If called later with the same arguments, the cached value is returned, and
not re-evaluated.
"""
def __init__(self, func):
self.func = func
self.cache = {}
def __call__(self, *args):
try:
return self.cache[args]
except KeyError:
value = self.func(*args)
self.cache[args] = value
return value
except TypeError:
# uncachable -- for instance, passing a list as an argument.
# Better to not cache than to blow up entirely.
return self.func(*args)
def __repr__(self):
"""Return the function's docstring."""
return self.func.__doc__
def __get__(self, obj, objtype):
"""Support instance methods."""
return functools.partial(self.__call__, obj)
@memoized
def fibonacci(n):
"Return the nth fibonacci number."
if n in (0, 1):
return n
return fibonacci(n-1) + fibonacci(n-2)
print fibonacci(12)
Properties do not automatically cache their return values. The getter (and setters) are intended to be called each time the property is accessed.
However, Denis Otkidach has written a wonderful cached attribute decorator (published in the Python Cookbook, 2nd edition and also originally on ActiveState under the PSF license) for just this purpose:
class cache(object):
'''Computes attribute value and caches it in the instance.
Python Cookbook (Denis Otkidach) https://stackoverflow.com/users/168352/denis-otkidach
This decorator allows you to create a property which can be computed once and
accessed many times. Sort of like memoization.
'''
def __init__(self, method, name=None):
# record the unbound-method and the name
self.method = method
self.name = name or method.__name__
self.__doc__ = method.__doc__
def __get__(self, inst, cls):
# self: <__main__.cache object at 0xb781340c>
# inst: <__main__.Foo object at 0xb781348c>
# cls: <class '__main__.Foo'>
if inst is None:
# instance attribute accessed on class, return self
# You get here if you write `Foo.bar`
return self
# compute, cache and return the instance's attribute value
result = self.method(inst)
# setattr redefines the instance's attribute so this doesn't get called again
setattr(inst, self.name, result)
return result
Here is an example demonstrating its use:
def demo_cache():
class Foo(object):
@cache
def bar(self):
print 'Calculating self.bar'
return 42
foo=Foo()
print(foo.bar)
# Calculating self.bar
# 42
print(foo.bar)
# 42
foo.bar=1
print(foo.bar)
# 1
print(Foo.bar)
# __get__ called with inst = None
# <__main__.cache object at 0xb7709b4c>
# Deleting `foo.bar` from `foo.__dict__` re-exposes the property defined in `Foo`.
# Thus, calling `foo.bar` again recalculates the value again.
del foo.bar
print(foo.bar)
# Calculating self.bar
# 42
demo_cache()
The decorator from Denis Otkidach mentioned in @unutbu’s answer was published in O’Reilly’s Python Cookbook. Unfortunately O’Reilly doesn’t specify any license for code examples – just as informal permission to reuse the code.
If you need a cached property decorator with a liberal license, you can use Ken Seehof‘s @cached_property from ActiveState code recipes. It’s explicitly published under the MIT license.
def cached_property(f):
"""returns a cached property that is calculated by function f"""
def get(self):
try:
return self._property_cache[f]
except AttributeError:
self._property_cache = {}
x = self._property_cache[f] = f(self)
return x
except KeyError:
x = self._property_cache[f] = f(self)
return x
return property(get)
Python 3.2 onwards offers a built-in decorator that you can use to create a LRU cache:
@functools.lru_cache(maxsize=128, typed=False)
Alternatively, if you’re using Flask / Werkzeug, there’s the @cached_property decorator.
For Django, try from django.utils.functional import cached_property
Note: Adding for the sake of completeness of available options.
No, property is not cached by default. However there are several options to get that behaviour, I would like to add one more to that:
I’ve had to look it up, since I had this same question.
The functools package from the standard library will be getting a cached_property decorator as well. Unfortunately, it’s only available from Python 3.8 (as of time of this post, it’s 3.8a0). The alternative to waiting is to use a custom one, such as this one as mentioned by 0xc0de) or Django’s, for now, then switch later:
from django.utils.functional import cached_property
# from functools import cached_property # Only 3.8+ :(
To anyone who might be reading this in 2020, this functionality is now available in the functools module as part of the standard library as of Python 3.8.
https://docs.python.org/dev/library/functools.html#functools.cached_property
Important to note, classes that define their own __dict__ (or do not define one at all) or use __slots__ might not work as expected. For example, NamedTuple and metaclasses.