mypy arg-type avoidance using decorator

Question

I have a class, which defines an optional attribute

class Stuff:
    config: Optional[Dict] = None

    def __init__(self):
        pass

At some point in my code I have a function that takes the config attribute as an input.

def method(config: Dict):
    """docstring"""
    print(type(config))
    print(config)

The config argument now has to be a Dict. One way to solve this is to check prior to calling method like

stuff = Stuff()
if not isinstance(stuff.config, dict):
    raise Exception("config needs to be a dict")
method(stuff.config)

This is OK by mypy. Now, suppose I would rather check this using a decorator;

def decorator(func: Callable) -> Any:
    def wrapper(*args: Any, **kwargs: Any) -> Any:
        if not args[0]:
            raise Exception("config may not be None")
        return func(*args, **kwargs)

    return wrapper

@decorator
def method(config: Dict):
...

Calling this would then be reduced to

stuff = Stuff()
method(stuff.config)

This works as intended in runtime, but using the decorator solution yields a mypy error: Argument 1 to "method" has incompatible type "Optional[Dict[Any, Any]]"; expected "Dict[Any, Any]" [arg-type]

How can I adjust the decorator solution to work with mypy?

Edit; corrected the call to method.

Asked By: user2181884

||

Source

Answer 1

How can I adjust the decorator solution to work with mypy?

You can’t.

You specified the type is Foo, so subsequent
attempts to supply Optional[Foo] won’t lint cleanly.

Mypy currently does some remarkable lexical analysis
and theorem proving, but it doesn’t delve far enough
into the code to agree with your perspective
on "this argument supplied is good enough".
One could propose a feature, and merge a pull request,
but that’s out-of-scope for this SO question.

Consider defining a new class MyDict:
which makes things easy on static analysis,
and which does the app-specific runtime
checking / enforcement that you require.

Answered By: J_H

Answer 2

Consider this much more specific decorator, with useful type hints:

def decorator(func: Callable[[dict], None]) -> Callable[[Optional[dict]], None]:
    def wrapper(d: Optional[dict]) -> None:
        if d is None:
            raise Exception("config may not be None")
        return func(d)

    return wrapper

Now, you can decorate method and change its static type, because mypy can see from the type hints on decorator alone that it changes the static type of method.

@decorator
def method(config: dict):
    ...


# No errors, even though method originally could not accept an argument
# of None
stuff = Stuff()
method(stuff.config)

The decorator explicitly turns a function of type Callable[[dict], None] into a function of type Callable[[Optional[dict]], None] that behaves the way you want.

So the problem with your decorator is that it does not provide the static typing necessary for mypy to understand what it will do to method.

Can we generalize the static typing in the decorator to allow it to be applied to more functions? Consider what we need, given what it does:

func should have type Callable[[T, ...], RV]. The only thing we want to say for certain is that its first argument is type T. The other arguments and the return type are arbitrary.
wrapper‘s first argument should be of type Optional[T], and we’ll use type narrowing in the body to "reduce" the value to a value of type T before passing it to func.
wrapper should return a value of the same type as the original caller. (That’s the best we can do; we can’t say it will return exactly the same value func returns.)

So let’s give it a try:

$ cat tmp.py
from typing import Optional, Callable, TypeVar, ParamSpec, Concatenate


T = TypeVar('T')
RV = TypeVar('RV')
P = ParamSpec('P')

def decorator(func: Callable[Concatenate[T, P], RV]) -> Callable[Concatenate[Optional[T], P], RV]:
    def wrapper(x: Optional[T], *args: P.args, **kwargs: P.kwargs) -> RV:
        if x is None:
            raise Exception("first argument may not be None")
        return func(x, *args, **kwargs)
    return wrapper


class Stuff:
    config: Optional[dict] = None

    def __init__(self):
        pass


@decorator
def method(config: dict):
    ...

stuff = Stuff()
method(stuff.config)

$ mypy tmp.py
Success: no issues found in 1 source file

Note the use of ParamSpec to capture the types of whatever arguments are passed to func, so that we can provide the same types to the wrapper. (And that this requires Python 3.10 or later, or the typing-extensions package.)

Answered By: chepner

mypy arg-type avoidance using decorator

Question:

Answers: