RecursionError when trying to implement both __getattribute__/__setattr__ and _getitem__/__setitem__ in python class

Question:

I’m running into the dreaded RecursionError while trying to implement a class that essentially acts as a proxy between a private variable _threads. The idea is for the class to act as a thread-aware dictionary, dishing out the correct dictionary assigned to a specific thread for each call to the class instance.

Background (Feel free to skip it if you don’t care)

I found myself needing this while parallelizing some reporting code that spans across many functions originally accessing many (consistent) instance variables. The class is basically a way to avoid having to grab the current thread identity within each function and manually doing the lookup in an instance dictionary. Plus, for re-usability in other projects I figured why not!

Current Code

import threading

class ThreadManager(object):
    """
    A custom thread-aware class for managing multiple sets of variables
    without the need to keep track of the current, working thread
    """

    def __init__(self, *args, **kwargs):
        assert threading.current_thread() is threading.main_thread(), 
            "ThreadManager must be instantiated from MainThread"
        self._threads = dict()
        self._thread_vars = dict(**dict.fromkeys(args), **kwargs)

    @property
    def thread(self) -> dict:
        ident = self.thread_id
        if ident not in self._threads:
            # new thread, create new key/value pair with default thread_vars dict as value
            self._threads[ident] = self._thread_vars
        return self._threads[ident]

    @property
    def thread_id(self) -> int:
        return threading.current_thread().ident

    def update(self, data: dict) -> None:
        ident = self.thread_id
        for var, value in data.items():
            if var not in self._thread_vars:
                raise AttributeError("%r was not found in the list of original thread variables" % var)
            self._threads[ident][var] = value

    def clear(self) -> None:
        """Clears a single thread's variables and re-instates the default values"""
        ident = self.thread_id
        self._threads[ident].clear()
        self._threads[ident].update(self._thread_vars)

    # region dunders
    def __setattr__(self, var, value):
        """
        >>> tm = ThreadManager("a", "b", c=False)
        >>> tm.b
        >>> tm.b = "data"
        >>> tm.b
        "data"
        """
        print(f"__setattr__: var={var}, value={value}")
        _ = self.thread  # FIXME: Hacky way to ensure the thread and accompanying dict exist before updating it
        self._threads[self.thread_id][var] = value

    def __getattr__(self, var):
        """
        >>> tm = ThreadManager("a", "b", c=False)
        >>> tm.b
        >>> tm.c
        False
        """
        print(f"__getattr__: var={var}")
        return self.thread[var]

    def __setitem__(self, var, value):
        """
        >>> tm = ThreadManager("a", "b", c=False)
        >>> tm["b"]
        >>> tm["b"] = "data"
        >>> tm["b"]
        "data"
        """
        print(f"__setitem__: var={var}, value={value}")
        _ = self.thread  # FIXME: Hacky way to ensure the thread and accompanying dict exist before updating it
        self._threads[self.thread_id][var] = value

    def __getitem__(self, var):
        """
        >>> tm = ThreadManager("a", "b", c=False)
        >>> tm["b"]
        >>> tm["c"]
        False
        """
        print(f"__getitem__: var={var}")
        return self.thread[var]

    def __len__(self):
        """Total number of threads being managed"""
        return len(self._threads)
    # endregion

if __name__ == "__main__":
    tm = ThreadManager("a", "b", c={"d": 1})

Using the above code I end up with the RecursionError and traceback:

__setattr__: var=_threads, value={}
__getattr__: var=_threads
__getattr__: var=_threads
__getattr__: var=_threads
...
__getattr__: var=_threads
__getattr__: var=_threads
Traceback (most recent call last):
  File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 1266, in <module>
    tm = ThreadManager("a", "b", c={"d": 1})
  File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 87, in __init__
    self._threads = dict()
  File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 125, in __setattr__
    _ = self.thread  # FIXME: Hacky way to ensure the thread and accompanying dict exist before updating it
  File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 93, in thread
    if ident not in self._threads:
  File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 136, in __getattr__
    return self.thread[var]
  File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 93, in thread
    if ident not in self._threads:
  File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 136, in __getattr__
    return self.thread[var]
  File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 93, in thread
    if ident not in self._threads:
  File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 136, in __getattr__
    return self.thread[var]
...
  File "C:/Users/mhill/PycharmProjects/reporting/app/reports/base.py", line 100, in thread_id
    return threading.current_thread().ident
  File "C:UsersmhillAppDataLocalProgramsPythonPython37libthreading.py", line 1233, in current_thread
    return _active[get_ident()]
RecursionError: maximum recursion depth exceeded while calling a Python object

I’m trying to keep the abilities to both access the dictionary items as properties (using tm.property_name) and as you would a typical dictionary item (using tm["property_name"]), which is why I’m trying to implement both __getattribute__/__setattr__ and __getitem__/__setitem__

Can anyone offer any insight on how I can resolve this issue?

Asked By: CaffeinatedMike

||

Answers:

You almost certainly want to use __getattr__ instead of __getattribute__. The former is called only when the normal instance variable lookup fails, but the latter is ALWAYS called. This is clearly explained in the documentation. It also seems to be exactly what you want here, unless I’m missing something. You want to be able to access certain variables in the normal way, for example thread and update, but you want to provide special functionality in other cases. That’s exactly what __getattr__ is for.

It is very difficult to implement __getattribute__ correctly, since you must handle all the variables that you want to access normally by explicitly calling __getattr__ in those cases. I don’t see any reason why you have to do this.

My suggestion is to just replace __getattribute__ by __getattr__ and try to run your program again.

Answered By: Paul Cornelius