Why are generics in Python implemented using __class_getitem instead of getitem__ on metaclass

Question

I was reading python documentation and peps and couldn’t find an answer for this.

Generics in python are implemented by subscripting class objects. list[str] is a list where all elements are strings.
This behaviour is achieved by implementing a special (dunder) classmethod called __class_getitem__ which as the documentation states should return a GenericAlias.

An example:

class MyGeneric:
    def __class_getitem__(cls, key):
        # implement generics
        ...

This seems weird to me because the documentation also shows some code similar to what the interpreter does when faced with subscripting objects and shows that defining both __getitem__ on object’s metaclass and __class_getitem__ on the object itself always chooses the metaclass’ __getitem__. This means that a class with the same functionality as the one above can be implemented without introducing a new special method into the language.

An example of a class with identical behaviour:

class GenericMeta(type):
    def __getitem__(self, key):
        # implement generics
        ...


class MyGeneric(metaclass=GenericMeta):
    ...

Later the documentation also shows an example of Enums using a __getitem__ of a metaclass as an example of a __class_getitem__ not being called.

My question is why was the __class_getitem__ classmethod introduced in the first place?

It seems to do the exact same thing as the metaclass’ __getitem__ but with the added complexity and the need for extra code in the interpreter for deciding which method to call. All of this comes with no extra benefit as defining both will simply call the same one every time unless specifically calling dunder methods (which should not be done in general).

I know that implementing generics this way is discouraged. The general approach is to subclass a class that already defines a __class_getitem__ like typing.Generic but I’m still curious as to why that functionality was implemented that way.

Asked By: user18769012

||

Source

Answer 1

__class_getitem__ exists because using multiple inheritance where multiple metaclasses are involved is very tricky and sets limitations that can’t always be met when using 3rd-party libraries.

Without __class_getitem__ generics requires a metaclass, as defining a __getitem__ method on a class would only handle attribute access on instances, not on the class. Normally, object[...] syntax is handled by the type of object, not by object itself. For instances, that’s the class, but for classes, that’s the metaclass.

So, the syntax:

ClassObject[some_type]

would translate to:

type(ClassObject).__getitem__(ClassObject, some_type)

__class_getitem__ exists to avoid having to give every class that needs to support generics, a metaclass.

For how __getitem__ and other special methods work, see the Special method lookup section in the Python Datamodel chapter:

For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object’s type, not in the object’s instance dictionary.

The same chapter also explicitly covers __class_getitem__ versus __getitem__:

Usually, the subscription of an object using square brackets will call the __getitem__() instance method defined on the object’s class. However, if the object being subscribed is itself a class, the class method __class_getitem__() may be called instead.

This section also covers what will happen if the class has both a metaclass with a __getitem__ method, and a __class_getitem__ method defined on the class itself. You found this section, but it only applies in this specific corner-case.

As stated, using metaclasses can be tricky, especially when inheriting from classes with different metaclasses. See the original PEP 560 – Core support for typing module and generic types proposal:

All generic types are instances of GenericMeta, so if a user uses a custom metaclass, then it is hard to make a corresponding class generic. This is particularly hard for library classes that a user doesn’t control.

…

With the help of the proposed special attributes the GenericMeta metaclass will not be needed.

When mixing multiple classes with different metaclasses, Python requires that the most specific metaclass derives from the other metaclasses, a requirement that can’t easily be met if the metaclass is not your own; see the documentation on determining the appropriate metaclass.

As a side note, if you do use a metaclass, then __getitem__ should not be a classmethod:

class GenericMeta(type):
    # not a classmethod! `self` here is a class, an instance of this
    # metaclass.
    def __getitem__(self, key):
        # implement generics
        ...

Before PEP 560, that’s basically what the typing.GenericMeta metaclass did, albeit with a bit more complexity.

Answered By: Martijn Pieters

Why are generics in Python implemented using __class_getitem instead of getitem__ on metaclass

Question:

Answers:

Why are generics in Python implemented using __class_getitem__ instead of __getitem__ on metaclass

Question:

Answers:

Why are generics in Python implemented using __class_getitem instead of getitem__ on metaclass