Python 3.7: check if type annotation is "subclass" of generic
Question:
I’m trying to find a reliable / cross-version (3.5+) way of checking whether a type annotation is a “subclass” of a given generic type (i.e. get the generic type out of the type annotation object).
On Python 3.5 / 3.6, it works a breeze, as you would expect:
>>> from typing import List
>>> isinstance(List[str], type)
True
>>> issubclass(List[str], List)
True
While on 3.7, it looks like instances of generic types are no longer instances of type, so it will fail:
>>> from typing import List
>>> isinstance(List[str], type)
False
>>> issubclass(List[str], List)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.7/typing.py", line 716, in __subclasscheck__
raise TypeError("Subscripted generics cannot be used with"
TypeError: Subscripted generics cannot be used with class and instance checks
Other ideas that come to mind are checking the actual instance type, but:
Python 3.6 / 3.5:
>>> type(List[str])
<class 'typing.GenericMeta'>
Python 3.7:
>>> type(List[str])
<class 'typing._GenericAlias'>
But that doesn’t really give any further indication as to which is the actual generic type (might not be List); besides, it feels quite wrong to be doing the check this way, especially since the _GenericAlias now became a “private” type (notice the underscore).
Another thing one could check is the __origin__ argument on the type, but that doesn’t feel like the right way to do it either.
And it still differs on 3.7:
>>> List[str].__origin__
<class 'list'>
while 3.5 / 3.6:
>>> List[str].__origin__
typing.List
I’ve been searching for the “right” way of doing this, but haven’t found it in the Python docs / google search.
Now, I’m assuming there must be a clean way of doing this check, as tools like mypy would rely on it for doing type checks..?
Update: about the use case
Ok adding a bit more context here..
So, my use case for this is using introspection on function signatures (argument types / defaults, return type, docstring) to automatically generate a GraphQL schema for them (thus reducing the amount of boilerplate).
I’m still a bit torn on whether this would be a good idea or not.
I like it from the usability point of view (no need to learn yet another way to declare your function signature: just annotate your types the usual way); see the two code examples here to understand what I mean: https://github.com/rshk/pyql
I wonder if supporting generic types (lists, dicts, unions, …) using types from typing this way adds too much “black magic”, that could break in unexpected ways. (It’s not a huge issue for now, but what about future Python versions, past 3.7? Is this going to become a maintenance nightmare?).
Of course the alternative would be to just use a custom type annotation that supports a more reliable / future-proof check, eg: https://github.com/rshk/pyql/blob/master/pyql/schema/types/core.py#L337-L339
..but on the downside, that would force people to remember they have to use the custom type annotation. Moreover, I’m not sure how would mypy deal with that (I assume there needs to be a declaration somewhere to say the custom type is fully compatible with typing.List..? Still sounds hackish).
(I’m mostly asking for suggestions on the two approaches, and most importantly any pros/cons of the two alternatives I might have missed. Hope this doesn’t become “too broad” for SO..).
Answers:
First of all: There is no API defined to introspect type hinting objects as defined by the typing module. Type hinting tools are expected to deal with source code, so text, not with Python objects at runtime; mypy doesn’t introspect List[str] objects, it instead deals a parsed Abstract Syntax Tree of your source code.
So, while you can always access attributes like __origin__, you are essentially dealing with implementation details (internal bookkeeping), and those implementation details can and will change from version to version.
That said, a core mypy / typing contributor has created the typing_inspect module to develop an introspection API for type hints. The project still documents itself as experimental, and you can expect that to change with time too until it isn’t experimental any more. It won’t solve your problem here, as it doesn’t support Python 3.5, and it’s get_origin() function returns the exact same values the __origin__ attribute provides.
With all those caveats out of the way, what you want to access on Python 3.5 / Python 3.6 is the __extra__ attribute; this is the base built-in type used to drive the issubclass() / isinstance() support that the library originally implemented (but since removed in 3.7):
def get_type_class(typ):
try:
# Python 3.5 / 3.6
return typ.__extra__
except AttributeError:
# Python 3.7
return typ.__origin__
This produces <class 'list'> in Python 3.5 and up, regardless. It still uses internal implementation details and may well break in future Python versions.
Python 3.8 adds typing.get_origin() and typing.get_args() to support basic introspection.
These APIs have also been backported to Python >=3.5 in https://pypi.org/project/typing-compat/.
Be aware that the behavior of typing.get_args is still subtly different in 3.7 when called on the bare generics; in 3.8 typing.get_args(typing.Dict) is (), but in 3.7 it is (~KT, ~VT) (and analogously for the other generics), where ~KT and ~VT are objects of type typing.TypeVar.
pip install typing_utils
Then
>>> typing_utils.issubtype(typing.List[int], list)
True
>>> typing_utils.issubtype(typing.List, typing.List[int])
False
typing_utils also backports typing.get_origin and typing.get_args from Python 3.8 to 3.6+.
I’m trying to find a reliable / cross-version (3.5+) way of checking whether a type annotation is a “subclass” of a given generic type (i.e. get the generic type out of the type annotation object).
On Python 3.5 / 3.6, it works a breeze, as you would expect:
>>> from typing import List
>>> isinstance(List[str], type)
True
>>> issubclass(List[str], List)
True
While on 3.7, it looks like instances of generic types are no longer instances of type, so it will fail:
>>> from typing import List
>>> isinstance(List[str], type)
False
>>> issubclass(List[str], List)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.7/typing.py", line 716, in __subclasscheck__
raise TypeError("Subscripted generics cannot be used with"
TypeError: Subscripted generics cannot be used with class and instance checks
Other ideas that come to mind are checking the actual instance type, but:
Python 3.6 / 3.5:
>>> type(List[str])
<class 'typing.GenericMeta'>
Python 3.7:
>>> type(List[str])
<class 'typing._GenericAlias'>
But that doesn’t really give any further indication as to which is the actual generic type (might not be List); besides, it feels quite wrong to be doing the check this way, especially since the _GenericAlias now became a “private” type (notice the underscore).
Another thing one could check is the __origin__ argument on the type, but that doesn’t feel like the right way to do it either.
And it still differs on 3.7:
>>> List[str].__origin__
<class 'list'>
while 3.5 / 3.6:
>>> List[str].__origin__
typing.List
I’ve been searching for the “right” way of doing this, but haven’t found it in the Python docs / google search.
Now, I’m assuming there must be a clean way of doing this check, as tools like mypy would rely on it for doing type checks..?
Update: about the use case
Ok adding a bit more context here..
So, my use case for this is using introspection on function signatures (argument types / defaults, return type, docstring) to automatically generate a GraphQL schema for them (thus reducing the amount of boilerplate).
I’m still a bit torn on whether this would be a good idea or not.
I like it from the usability point of view (no need to learn yet another way to declare your function signature: just annotate your types the usual way); see the two code examples here to understand what I mean: https://github.com/rshk/pyql
I wonder if supporting generic types (lists, dicts, unions, …) using types from typing this way adds too much “black magic”, that could break in unexpected ways. (It’s not a huge issue for now, but what about future Python versions, past 3.7? Is this going to become a maintenance nightmare?).
Of course the alternative would be to just use a custom type annotation that supports a more reliable / future-proof check, eg: https://github.com/rshk/pyql/blob/master/pyql/schema/types/core.py#L337-L339
..but on the downside, that would force people to remember they have to use the custom type annotation. Moreover, I’m not sure how would mypy deal with that (I assume there needs to be a declaration somewhere to say the custom type is fully compatible with typing.List..? Still sounds hackish).
(I’m mostly asking for suggestions on the two approaches, and most importantly any pros/cons of the two alternatives I might have missed. Hope this doesn’t become “too broad” for SO..).
First of all: There is no API defined to introspect type hinting objects as defined by the typing module. Type hinting tools are expected to deal with source code, so text, not with Python objects at runtime; mypy doesn’t introspect List[str] objects, it instead deals a parsed Abstract Syntax Tree of your source code.
So, while you can always access attributes like __origin__, you are essentially dealing with implementation details (internal bookkeeping), and those implementation details can and will change from version to version.
That said, a core mypy / typing contributor has created the typing_inspect module to develop an introspection API for type hints. The project still documents itself as experimental, and you can expect that to change with time too until it isn’t experimental any more. It won’t solve your problem here, as it doesn’t support Python 3.5, and it’s get_origin() function returns the exact same values the __origin__ attribute provides.
With all those caveats out of the way, what you want to access on Python 3.5 / Python 3.6 is the __extra__ attribute; this is the base built-in type used to drive the issubclass() / isinstance() support that the library originally implemented (but since removed in 3.7):
def get_type_class(typ):
try:
# Python 3.5 / 3.6
return typ.__extra__
except AttributeError:
# Python 3.7
return typ.__origin__
This produces <class 'list'> in Python 3.5 and up, regardless. It still uses internal implementation details and may well break in future Python versions.
Python 3.8 adds typing.get_origin() and typing.get_args() to support basic introspection.
These APIs have also been backported to Python >=3.5 in https://pypi.org/project/typing-compat/.
Be aware that the behavior of typing.get_args is still subtly different in 3.7 when called on the bare generics; in 3.8 typing.get_args(typing.Dict) is (), but in 3.7 it is (~KT, ~VT) (and analogously for the other generics), where ~KT and ~VT are objects of type typing.TypeVar.
pip install typing_utils
Then
>>> typing_utils.issubtype(typing.List[int], list)
True
>>> typing_utils.issubtype(typing.List, typing.List[int])
False
typing_utils also backports typing.get_origin and typing.get_args from Python 3.8 to 3.6+.