adding __getitem__ accessor to Python class method

Question:

I’m attempting to add an item getter (__getitem__, to provide the [] syntax) to a class method so that I can use some unique-ish syntax to provide types to functions outside the normal parentheses, like the following. The syntax on the last line (of this first snippet) is really the goal for this whole endeavor.

class MyClass:

    @typedmethod
    def compute_typed_value(self, value, *args, **kwargs):
        print(self, args, kwargs)
        result = TypedMethod.requested_type(kwargs)(value)
        self.my_other_method()
        return result

    def my_other_method(self):
        print('Doing some other things!')
        return 3


a = MyClass()
a.compute_typed_value[int]('12345') # returns int value 12345

Additionally, I’d like to retain the intuitive behavior that a defined function can be called like a function, potentially with a default value for the type, like so:

a = MyClass()
a.compute_typed_value('12345') 
# should return whatever the default type is, with the value of '12345', 
# or allow some other default behavior

In a broader context, this would be implemented as a piece of an API adapter that implements a generic request processor, and I’d like the data to come out of the API adapter in a specific format. So the way that this might look in actual use could be something like the following:


@dataclass
class MyAPIData:
    property_a: int = 0
    property_b: int = 0

class MyAPIAdapter:
    _session
    def __init__(self, token):
        self._init_session(token)

    @typedmethod
    def request_json(self, url, **kwargs):
        datatype = TypedMethod.requested_type(kwargs)
        response_data = self._session.get(url).json()
        if datatype:
            response_data = datatype(**response_data)
        return response_data

    def fetch_myapidata(self, search):
        return self.request_json[MyAPIData](f"/myapi?q={search}")

I’m attempting to achieve this kind of behavior with a decorator that I can throw onto any function that I want to enable this behavior. Here is my current full implementation:


from functools import partial

class TypedMethod:

    _REQUESTED_TYPE_ATTR = '__requested_type'

    def __init__(self, method):
        self._method = method
        print(method)
        self.__call__ = method.__call__

    def __getitem__(self, specified_type, *args, **kwargs):
        print(f'getting typed value: {specified_type}')
        if not isinstance(specified_type, type):
            raise TypeError("Only Type Accessors are supported - must be an instance of `type`")
            
        return partial(self.__call__, **{self.__class__._REQUESTED_TYPE_ATTR: specified_type})
    
    def __call__(self, *args, **kwargs):
        print(args, kwargs)
        return self._method(self, *args, **kwargs)
    
    @classmethod
    def requested_type(cls, foo_kwargs):
        return foo_kwargs[cls._REQUESTED_TYPE_ATTR] if cls._REQUESTED_TYPE_ATTR in foo_kwargs else None

def typedmethod(foo):
    print(f'wrapping {foo.__name__} with a Typed Method: {foo}')
    _typed_method = TypedMethod(foo)
    def wrapper(self, *args, **kwargs):
        print('WRAPPER', self, args, kwargs)
        return _typed_method(self, *args, **kwargs)
    _typed_method.__call__ = wrapper
    return _typed_method

class MyClass:

    @typedmethod
    def compute_typed_value(self, value, *args, **kwargs):
        print(self, args, kwargs)
        result = TypedMethod.requested_type(kwargs)(value)
        print(result)
        self.my_other_method()
        return result

    def my_other_method(self):
        print('Doing some other things!')
        return 3


a = MyClass()
a.compute_typed_value[int]('12345')

If you run this code, it will fail stating that ‘TypedMethod’ object has no attribute ‘my_other_method’. Further inspection reveals that the first line of compute_typed_value is not printing what one would intuitively expect from the code:

<__main__.TypedMethod object at 0x10754e790> () {'__requested_type': <class 'int'>}

Specifically, the first item printed, which is a TypedMethod instead of a MyClass instance

Basically, the idea is use the __getitem__ callout to generate a functools.partial so that the subsequent call to the resulting function contains the __getitem__ key in a known "magic" kwargs value, which should hypothetically work, except that now the self reference that is available to MyClass.compute_typed_value is actually a reference to the TypedMethod instance generated by the wrapper instead of the expected MyClass instance. I’ve attempted a number of things to get the MyClass instance passed as self, but since it’s implemented as a decorator, the instance isn’t available at the time of decoration, meaning that somehow it needs to be a bound method at the time of function execution, I think.


I know I could just pass this value in as like the first positional argument, but I want it to work with the square bracket annotation because I think it’d be cool and more readable. This is mostly a learning exercise to understand more of Python’s inner workings, so the answer could ultimately be "no".

Asked By: David Culbreth

||

Answers:

We can make SomeInstance class with a getter that returns MyClass‘s instance by invoking the descriptor and setting it as a class attribute when init‘ing MyClass. Here MyClass._instance is the instance of MyClass (it’s MyClass‘s self), which we can pass to the instance method my_other_method within the decorated computed_typed_value method:

class SomeInstance:
    def __get__(self, instance, owner):
        if instance is None:
            print('instance is None')
            return self
        print(f'getting instance of {owner.__name__} via {self.__class__.__name__}')
        return instance

class MyClass:
    _instance = SomeInstance()
    def __init__(self):
        MyClass._instance = self._instance
        print(f'MyClass._instance type is {type(MyClass._instance)}')

    def my_other_method(self):
        print('Doing some other things!')
        return 3

    @typedmethod
    def compute_typed_value(self, value, *args, **kwargs):
        print(self, args, kwargs)
        result = TypedMethod.requested_type(kwargs)(value)
        print(result)
        x = MyClass._instance.my_other_method()
        print(x)
        return result

a = MyClass()
a.compute_typed_value[int]('12345')

Outputs:

wrapping compute_typed_value with a Typed Method: <function MyClass.compute_typed_value at 0x7f5dce1ed790>
<function MyClass.compute_typed_value at 0x7f5dce1ed790>
getting instance of MyClass via SomeInstance
MyClass._instance type is <class '__main__.MyClass'>
getting typed value: <class 'int'>
WRAPPER 12345 () {'__requested_type': <class 'int'>}
('12345',) {'__requested_type': <class 'int'>}
<__main__.TypedMethod object at 0x7f5deed5a7c0> () {'__requested_type': <class 'int'>}
12345
Doing some other things!
3
12345
Answered By: Ori Yarden

Your code is doing some odd stuff with __call__ that doesn’t quite work. Fixing those issues will likely make self refer to what you expect in compute_typed_value.

The main problems:

  • Assigning a new function to the __call__ attribute of an instance doesn’t work to change the object’s behavior when it’s actually called. You attempt this twice, but the TypedMethod object’s hard-coded __call__ method is getting called instead of any of the other things you try (you first set _method.__call__ and separately wrapper to be called, neither of which make much sense to me).
  • Your typed_method decorator returns the TypedMethod object it creates, rather than the wrapper function. Because TypedMethod is not a descriptor, there’s no binding logic for MyClass.compute_typed_value, so there’s no good way for the instance of MyClass to get passed in anywhere. Normally this works because functions are descriptors, returning bound method objects. However, it’s going to be a bit complicated to make that work here, since you want a __getattr__ to work on the bound object.

So, I think you should change things up to use two different classes.

The first is a descriptor class, that when looked up, has binding behavior so that you can get the self value to pass in to the method. When bound, it returns an instance of the second class.

The second class handles the indexing by type. It has a __getitem__ method, which returns a partial that passes both the self value that the first class captured, and the type that it has been indexed with (as a secret keyword argument).

Here’s what that looks like:

class typedmethod:
    def __init__(self, method):
        self.method = method

    def __get__(self, instance, owner=None):
        if instance is None: return self # class lookup
        return TypeIndexer(instance, self.method)

class TypeIndexer:
    def __init__(self, instance, method):
        self.instance = instance
        self.method = method

    def __getitem__(self, type):
        return partial(method, self.instance, _secret_kwarg=type)

I’ve left out the logic to hide the name _secret_kwarg in a class variable somewhere, and to have a public API for getting it out of a kwargs dict. It would actually be a whole lot easier if you just passed the type in to the method as a public argument. Maybe make it the first positional arguement after self, or a kwarg with a meaningful name? The fact that the user doesn’t actually supply it directly wouldn’t be much more confusing than TypedMethod.requested_type(kwargs)(value) is now.

Of course, if we follow that logic to its conclusion, you could rewrite the whole obj.method[type](args) pattern to be obj.method(type, args) and it would be a whole lot easier.

Answered By: Blckknght

Applying your concept to a class has a giant roadblock. This is what the decorator does:

    def typewrap(func):
        def wrapper(self, *args, **kwargs):
            #...do type checking

            #call decorated method
            func(self, *args, **kwargs)
            
        #there is no `self` of `func` to store
        #and scope is changing
        #`self` of `func` is abandoned here
        return TypedMethod(wrapper)

You need a reference to self of func that you can pass to TypedMethod, before wrapper is run, but it doesn’t exist yet. It also won’t exist, because wrapper will be called from the TypedMethod instance, instead of your class instance.


If you get rid of the class scope, your idea becomes very simple to implement. A bunch of your original logic is not necessary (mainly __call__). Here is an example:

from functools import partial

class TypedMethod:
    def __init__(self, callback):
        self._cb = callback

    def __getitem__(self, key):
        if not isinstance(key, type):
            raise TypeError("key must be an instance of `type`")
            
        return partial(self._cb, key)
       
       
def typedmethod(func):
    def wrapper(argtype:type, *args, **kwargs):
        #test all arguments
        for arg in args:
            if not isinstance(arg, argtype):
                raise TypeError(f"all arguments must be of type: {argtype.__name__}")
        for _,v in kwargs.items():
            if not isinstance(v, argtype):
                raise TypeError(f"all arguments must be of type: {argtype.__name__}")
                
        #call wrapped method
        func(*args, **kwargs)
        
    #assign wrapper as the callback
    return TypedMethod(wrapper)
def dofunc(a):
    print(a)
    
@typedmethod
def dotype(x,y,z,a):
    print(x,y,z)
    dofunc(a)

dotype[int](22, 35, 2, a=12345)
#22, 35, 2
#12345
Answered By: OneMadGypsy

I never thought I would use a Python class this way, but if the pattern fits…

The issue I was having was that the class instance reference (self) was being overridden due to the reassignment of the function itself via a decorator. My solution is basically to add another class-level decorator to manually put that reference back. This looks a little different than my originally asked question due to a change in naming, but the essence is still the same. We define a class that is itself used as the decorator. Python just allows this because class types, when called, are function calls themselves, to create and init a new instance of the class. This also allows us to provide a few more magic methods to make other quality of life items easier. Here’s what that looks like:

from functools import partial, wraps
from types import MethodType

class subscriptable:
    _SUBSCRIPT_KEY = '___SUBSCRIPT_KEY'
    _HAS_SUBSCRIPTABLE_METHODS = '___HAS_SUBSCRIPTABLE_METHODS'

    def __init__(self, method):
        self._method = method

    def _bind(self, instance):
        self._method = MethodType(self._method, instance)

    def __getitem__(self, specified_type, *args, **kwargs):
        if not isinstance(specified_type, type):
            raise TypeError("Only Type Accessors are supported - must be an instance of `type`")
            
        return partial(self.__call__, **{self.__class__._SUBSCRIPT_KEY: specified_type})
    
    def __call__(self, *args, **kwargs):
        """A transparent passthrough to the wrapped method"""
        return self._method(*args, **kwargs)
    
    def __str__(self):
        return f"<{self.__class__.__name__} {self._method}>"
    
    @classmethod
    def has_key(cls, foo_kwargs):
        """A utility method to determine whether the provided kwargs has the expected subscript key"""
        return cls._SUBSCRIPT_KEY in foo_kwargs
    
    @classmethod
    def key(cls, foo_kwargs:dict):
        """A utility method that allows the subscript key to be consumed by the wrapped method, without needing to know the inner workings"""
        return foo_kwargs.pop(cls._SUBSCRIPT_KEY, None)
    
    @classmethod
    def container(cls, clazz):
        """A decorator for classes containing `subscriptable` methods"""
        if not hasattr(clazz, cls._HAS_SUBSCRIPTABLE_METHODS):
            orig_init = clazz.__init__
            @wraps(clazz.__init__)
            def __init__(self, *args, **kwargs):
                for attr_name in dir(self):
                    attr_value = getattr(self, attr_name)
                    if isinstance(attr_value, cls):
                        attr_value._bind(self)
                orig_init(self, *args, **kwargs)
            clazz.__init__ = __init__
            setattr(clazz, cls._HAS_SUBSCRIPTABLE_METHODS, True)
        return clazz

As a bonus feature, As some previously written answers allude to, class functions are not the only place that something like this might be useful, so this also allows for standalone functions to exhibit the same behavior. See this example, and its output:


@subscriptable
def other_typed_value(value, **kwargs):
    subscript = subscriptable.key(kwargs)
    print(subscript, value)

value = other_typed_value[int]('12345')
value = other_typed_value('12345')
print("Standard function str:", other_typed_value)

Produces the output:

<class 'int'> 12345
None 12345
Standard function str: <subscriptable <function other_typed_value at 0x000001DC809384C0>>

And finally, the original point of the question, whether we can apply this pattern to class methods. The answer is yes, but with the assistance of yet another decorator. This is where subscriptable.container steps in. Since we can’t access the instance at the time of class definition, I used an additional decorator to provide a pre-init hook that initializes all the functions so they are usable as expected (as properly bound class methods, even!), available even in the __init__ function. This kind of processing is probably pretty slow, but for my use case, it’s mostly for singletons anyway.

@subscriptable.container
class MyClass:

    @subscriptable
    def compute_typed_value(self, value, *args, **kwargs):
        print(self, args, kwargs)
        if subscriptable.has_key(kwargs):
            value = subscriptable.key(kwargs)(value)
        self.my_other_method()
        return value

    def my_other_method(self):
        print('Doing some other things!')
        return 3


a = MyClass()
value = a.compute_typed_value[int]('12345')
print(value, type(value))
value = a.compute_typed_value('12345')
print(value, type(value))
print("Class Method str:", a.compute_typed_value)

Anyhoo, the above code yields the following output, which you’ll notice has all the correct references in the places they were missing before. great success!

Doing some other things!
12345 <class 'int'>
<__main__.MyClass object at 0x000001DC808EB1C0> () {}
Doing some other things!
12345 <class 'str'>
Class Method str: <subscriptable <bound method MyClass.compute_typed_value of <__main__.MyClass object at 0x000001DC808EB1C0>>>

I was hoping to do this without a second decorator, but a single class decorator to enable the desired behavior when I’m already using one is a price I’m willing to pay.

Answered By: David Culbreth

In response to this answer. There are a number of issues.

Firstly, none of the below code is necessary, nor any other spot that uses it. It’s just an arbitrary layer that’s in the way.


class subscriptable:
    _parent_instance = None
    _SUBSCRIPT_KEY = '___SUBSCRIPT_KEY'
    _HAS_SUBSCRIPTABLE_METHODS = '___HAS_SUBSCRIPTABLE_METHODS'

    def __str__(self):
        return f"<{self.__class__.__name__} {self._method}>"
    
    @classmethod
    def has_key(cls, foo_kwargs):
        """A utility method to determine whether the provided kwargs has the expected subscript key"""
        return cls._SUBSCRIPT_KEY in foo_kwargs
    
    @classmethod
    def key(cls, foo_kwargs:dict):
        """A utility method that allows the subscript key to be consumed by the wrapped method, without needing to know the inner workings"""
        return foo_kwargs.pop(cls._SUBSCRIPT_KEY, None)

Secondly, there is no place in any of your code that is actually doing anything regarding the type you set. I’m going to assume that you want the method/function to only accept that type. If so:

class subscriptable:
    def __init__(self, callback:callable):
        self._cb = callback
        
    def __post_init__(self, parent):
        self._cb = MethodType(self._cb, parent)
             
    def __getitem__(self, argtype:type, *args, **kwargs):
        if not isinstance(argtype, type):
            raise TypeError("key must be an instance of `type`")
            
        return partial(self.__call__, argtype)
    
    def __call__(self, argtype:type, *args, **kwargs):
        #check all arguments against type
        for arg in args:
            if not isinstance(arg, argtype):
                raise TypeError(f"all arguments must be of type: {argtype.__name__}")

        for _,v in kwargs.items():
            if not isinstance(v, argtype):
                raise TypeError(f"all arguments must be of type: {argtype.__name__}")
         
        return self._cb(*args, **kwargs)
    
    @classmethod
    def container(cls, C):
        init = C.__init__
        @wraps(init)
        def __init__(self, *args, **kwargs):
            for name in dir(self):
                X = getattr(self, name)
                if isinstance(X, cls):
                    X.__post_init__(self)
            init(self, *args, **kwargs)
        C.__init__ = __init__
        return C

@subscriptable.container        
class MyClass:
    @subscriptable
    def compute_typed_value(self, *args, **kwargs):
        print(self, args, kwargs)
        self.my_other_method()

    def my_other_method(self):
        print('Doing some other things!')

value = MyClass().compute_typed_value[int](12345)
@subscriptable
def other_typed_value(*args, **kwargs):
    print(args, kwargs)

value = other_typed_value[int](12345)

I never thought I would use a Python class this way, but if the
pattern fits…

You’re wrapping classes with methods and methods with classes to solve a problem that you created. What pattern does this fit?

Answered By: OneMadGypsy