RecursionError when trying to implement both __getattribute__/__setattr__ and _getitem__/__setitem__ in python class
Question:
I’m running into the dreaded RecursionError
while trying to implement a class that essentially acts as a proxy between a private variable _threads
. The idea is for the class to act as a thread-aware dictionary, dishing out the correct dictionary assigned to a specific thread for each call to the class instance.
Background (Feel free to skip it if you don’t care)
I found myself needing this while parallelizing some reporting code that spans across many functions originally accessing many (consistent) instance variables. The class is basically a way to avoid having to grab the current thread identity within each function and manually doing the lookup in an instance dictionary. Plus, for re-usability in other projects I figured why not!
Current Code
import threading
class ThreadManager(object):
"""
A custom thread-aware class for managing multiple sets of variables
without the need to keep track of the current, working thread
"""
def __init__(self, *args, **kwargs):
assert threading.current_thread() is threading.main_thread(),
"ThreadManager must be instantiated from MainThread"
self._threads = dict()
self._thread_vars = dict(**dict.fromkeys(args), **kwargs)
@property
def thread(self) -> dict:
ident = self.thread_id
if ident not in self._threads:
# new thread, create new key/value pair with default thread_vars dict as value
self._threads[ident] = self._thread_vars
return self._threads[ident]
@property
def thread_id(self) -> int:
return threading.current_thread().ident
def update(self, data: dict) -> None:
ident = self.thread_id
for var, value in data.items():
if var not in self._thread_vars:
raise AttributeError("%r was not found in the list of original thread variables" % var)
self._threads[ident][var] = value
def clear(self) -> None:
"""Clears a single thread's variables and re-instates the default values"""
ident = self.thread_id
self._threads[ident].clear()
self._threads[ident].update(self._thread_vars)
# region dunders
def __setattr__(self, var, value):
"""
>>> tm = ThreadManager("a", "b", c=False)
>>> tm.b
>>> tm.b = "data"
>>> tm.b
"data"
"""
print(f"__setattr__: var={var}, value={value}")
_ = self.thread # FIXME: Hacky way to ensure the thread and accompanying dict exist before updating it
self._threads[self.thread_id][var] = value
def __getattr__(self, var):
"""
>>> tm = ThreadManager("a", "b", c=False)
>>> tm.b
>>> tm.c
False
"""
print(f"__getattr__: var={var}")
return self.thread[var]
def __setitem__(self, var, value):
"""
>>> tm = ThreadManager("a", "b", c=False)
>>> tm["b"]
>>> tm["b"] = "data"
>>> tm["b"]
"data"
"""
print(f"__setitem__: var={var}, value={value}")
_ = self.thread # FIXME: Hacky way to ensure the thread and accompanying dict exist before updating it
self._threads[self.thread_id][var] = value
def __getitem__(self, var):
"""
>>> tm = ThreadManager("a", "b", c=False)
>>> tm["b"]
>>> tm["c"]
False
"""
print(f"__getitem__: var={var}")
return self.thread[var]
def __len__(self):
"""Total number of threads being managed"""
return len(self._threads)
# endregion
if __name__ == "__main__":
tm = ThreadManager("a", "b", c={"d": 1})
Using the above code I end up with the RecursionError
and traceback:
__setattr__: var=_threads, value={}
__getattr__: var=_threads
__getattr__: var=_threads
__getattr__: var=_threads
...
__getattr__: var=_threads
__getattr__: var=_threads
Traceback (most recent call last):
File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 1266, in <module>
tm = ThreadManager("a", "b", c={"d": 1})
File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 87, in __init__
self._threads = dict()
File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 125, in __setattr__
_ = self.thread # FIXME: Hacky way to ensure the thread and accompanying dict exist before updating it
File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 93, in thread
if ident not in self._threads:
File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 136, in __getattr__
return self.thread[var]
File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 93, in thread
if ident not in self._threads:
File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 136, in __getattr__
return self.thread[var]
File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 93, in thread
if ident not in self._threads:
File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 136, in __getattr__
return self.thread[var]
...
File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 100, in thread_id
return threading.current_thread().ident
File "C:UsersmhillAppDataLocalProgramsPythonPython37libthreading.py", line 1233, in current_thread
return _active[get_ident()]
RecursionError: maximum recursion depth exceeded while calling a Python object
I’m trying to keep the abilities to both access the dictionary items as properties (using tm.property_name
) and as you would a typical dictionary item (using tm["property_name"]
), which is why I’m trying to implement both __getattribute__/__setattr__
and __getitem__/__setitem__
Can anyone offer any insight on how I can resolve this issue?
Answers:
You almost certainly want to use __getattr__
instead of __getattribute__
. The former is called only when the normal instance variable lookup fails, but the latter is ALWAYS called. This is clearly explained in the documentation. It also seems to be exactly what you want here, unless I’m missing something. You want to be able to access certain variables in the normal way, for example thread
and update
, but you want to provide special functionality in other cases. That’s exactly what __getattr__
is for.
It is very difficult to implement __getattribute__
correctly, since you must handle all the variables that you want to access normally by explicitly calling __getattr__
in those cases. I don’t see any reason why you have to do this.
My suggestion is to just replace __getattribute__
by __getattr__
and try to run your program again.
I’m running into the dreaded RecursionError
while trying to implement a class that essentially acts as a proxy between a private variable _threads
. The idea is for the class to act as a thread-aware dictionary, dishing out the correct dictionary assigned to a specific thread for each call to the class instance.
Background (Feel free to skip it if you don’t care)
I found myself needing this while parallelizing some reporting code that spans across many functions originally accessing many (consistent) instance variables. The class is basically a way to avoid having to grab the current thread identity within each function and manually doing the lookup in an instance dictionary. Plus, for re-usability in other projects I figured why not!
Current Code
import threading
class ThreadManager(object):
"""
A custom thread-aware class for managing multiple sets of variables
without the need to keep track of the current, working thread
"""
def __init__(self, *args, **kwargs):
assert threading.current_thread() is threading.main_thread(),
"ThreadManager must be instantiated from MainThread"
self._threads = dict()
self._thread_vars = dict(**dict.fromkeys(args), **kwargs)
@property
def thread(self) -> dict:
ident = self.thread_id
if ident not in self._threads:
# new thread, create new key/value pair with default thread_vars dict as value
self._threads[ident] = self._thread_vars
return self._threads[ident]
@property
def thread_id(self) -> int:
return threading.current_thread().ident
def update(self, data: dict) -> None:
ident = self.thread_id
for var, value in data.items():
if var not in self._thread_vars:
raise AttributeError("%r was not found in the list of original thread variables" % var)
self._threads[ident][var] = value
def clear(self) -> None:
"""Clears a single thread's variables and re-instates the default values"""
ident = self.thread_id
self._threads[ident].clear()
self._threads[ident].update(self._thread_vars)
# region dunders
def __setattr__(self, var, value):
"""
>>> tm = ThreadManager("a", "b", c=False)
>>> tm.b
>>> tm.b = "data"
>>> tm.b
"data"
"""
print(f"__setattr__: var={var}, value={value}")
_ = self.thread # FIXME: Hacky way to ensure the thread and accompanying dict exist before updating it
self._threads[self.thread_id][var] = value
def __getattr__(self, var):
"""
>>> tm = ThreadManager("a", "b", c=False)
>>> tm.b
>>> tm.c
False
"""
print(f"__getattr__: var={var}")
return self.thread[var]
def __setitem__(self, var, value):
"""
>>> tm = ThreadManager("a", "b", c=False)
>>> tm["b"]
>>> tm["b"] = "data"
>>> tm["b"]
"data"
"""
print(f"__setitem__: var={var}, value={value}")
_ = self.thread # FIXME: Hacky way to ensure the thread and accompanying dict exist before updating it
self._threads[self.thread_id][var] = value
def __getitem__(self, var):
"""
>>> tm = ThreadManager("a", "b", c=False)
>>> tm["b"]
>>> tm["c"]
False
"""
print(f"__getitem__: var={var}")
return self.thread[var]
def __len__(self):
"""Total number of threads being managed"""
return len(self._threads)
# endregion
if __name__ == "__main__":
tm = ThreadManager("a", "b", c={"d": 1})
Using the above code I end up with the RecursionError
and traceback:
__setattr__: var=_threads, value={}
__getattr__: var=_threads
__getattr__: var=_threads
__getattr__: var=_threads
...
__getattr__: var=_threads
__getattr__: var=_threads
Traceback (most recent call last):
File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 1266, in <module>
tm = ThreadManager("a", "b", c={"d": 1})
File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 87, in __init__
self._threads = dict()
File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 125, in __setattr__
_ = self.thread # FIXME: Hacky way to ensure the thread and accompanying dict exist before updating it
File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 93, in thread
if ident not in self._threads:
File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 136, in __getattr__
return self.thread[var]
File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 93, in thread
if ident not in self._threads:
File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 136, in __getattr__
return self.thread[var]
File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 93, in thread
if ident not in self._threads:
File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 136, in __getattr__
return self.thread[var]
...
File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 100, in thread_id
return threading.current_thread().ident
File "C:UsersmhillAppDataLocalProgramsPythonPython37libthreading.py", line 1233, in current_thread
return _active[get_ident()]
RecursionError: maximum recursion depth exceeded while calling a Python object
I’m trying to keep the abilities to both access the dictionary items as properties (using tm.property_name
) and as you would a typical dictionary item (using tm["property_name"]
), which is why I’m trying to implement both __getattribute__/__setattr__
and __getitem__/__setitem__
Can anyone offer any insight on how I can resolve this issue?
You almost certainly want to use __getattr__
instead of __getattribute__
. The former is called only when the normal instance variable lookup fails, but the latter is ALWAYS called. This is clearly explained in the documentation. It also seems to be exactly what you want here, unless I’m missing something. You want to be able to access certain variables in the normal way, for example thread
and update
, but you want to provide special functionality in other cases. That’s exactly what __getattr__
is for.
It is very difficult to implement __getattribute__
correctly, since you must handle all the variables that you want to access normally by explicitly calling __getattr__
in those cases. I don’t see any reason why you have to do this.
My suggestion is to just replace __getattribute__
by __getattr__
and try to run your program again.