Using typeguard decorator: @typechecked in Python, whilst evading circular imports?

Question:

Context

To prevent circular imports in Python when using type-hints, one can use the following construct:

# controllers.py
from __future__ import annotations
from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from models import Book


class BookController:
    def __init__(self, book: "Book") -> None:
        self.book = book

Where the if TYPE_CHECKING: is only executed during type checking, and not during execution of the code.

Issue

When one applies active function argument type verification, (based on the type hints of the arguments), typeguard throws the error:

NameError: name ‘Supported_experiment_settings’ is not defined

MWE I

# models.py
from controllers import BookController

from typeguard import typechecked

class Book:
    
    @typechecked
    def get_controller(self, some_bookcontroller:BookController):
        return some_bookcontroller

some_book=Book()
BookController("somestring")

And:

# controllers.py
from __future__ import annotations
from typing import TYPE_CHECKING
from typeguard import typechecked
#from models import Book

if TYPE_CHECKING:
    from models import Book

class BookController:
    
    @typechecked
    def __init__(self, book: Book) -> None:
        self.book = book

Note the #from models import Book is commented out. Now if one runs:

python models.py

It throws the error:

File "/home/name/Documents/eg/models.py", line 13, in
BookController("somestring")
…
NameError: name ‘Book’ is not defined. Did you mean: ‘bool’?
because the typechecking for def __init__(self, book: Book) -> None: does not know what the class Book is.

MWE II

Then if one disables @typechecked in controllers.py with:

# controllers.py
from __future__ import annotations
from typing import TYPE_CHECKING
from typeguard import typechecked

if TYPE_CHECKING:
    from models import Book

class BookController:
    
    #@typechecked
    def __init__(self, book: Book) -> None:
        self.book = book

it works. (But no typechecking).

MWE III

Then if one re-enables typechecking, and includes the import of book, (with from models import Book) like:

# controllers.py
from __future__ import annotations
from typing import TYPE_CHECKING
from typeguard import typechecked
from models import Book

if TYPE_CHECKING:
    from models import Book

class BookController:
    
    @typechecked
    def __init__(self, book: Book) -> None:
        self.book = book

It throws the circular import error:

Traceback (most recent call last):
  File "/home/name/Documents/eg/models.py", line 2, in <module>
    from controllers import BookController
  File "/home/name/Documents/eg/controllers.py", line 5, in <module>
    from models import Book
  File "/home/name/Documents/eg/models.py", line 2, in <module>
    from controllers import BookController
ImportError: cannot import name 'BookController' from partially initialized module 'controllers' (most likely due to a circular import) (/home/name/Documents/eg/controllers.py)

Question

How can one evade this circular import whilst still allowing the @typechecked decorator to verify/access the Book import?

Is there an equivalent TYPE_CHECKING boolean for typeguard?

Asked By: a.t.

||
Source

Answers:

The suggestion by @ShadowRanger worked. importing the module and only calling book in the typing, evaded the circular import (only when calling python controllers.py (which also runs models.py), though not when calling python models.py, (see Limitations)).

MWE

# models.py
import controllers

from typeguard import typechecked

class Book:
    
    @typechecked
    def get_controller(self, some_bookcontroller:controllers.BookController):
        return some_bookcontroller

some_book=Book()

controllers.BookController(some_book)
#controllers.BookController("somestring")

# controllers.py
from __future__ import annotations
from typing import TYPE_CHECKING
from typeguard import typechecked
import models

class BookController:
    
    @typechecked
    def __init__(self, book: models.Book) -> None:
        self.book = book

This can be ran with:

python controllers.py

and it throws a circular import error with:

python models.py

Error message:

Traceback (most recent call last):
  File "/home/name/Documents/eg/models.py", line 2, in <module>
    import controllers
  File "/home/name/Documents/eg/controllers.py", line 5, in <module>
    import models
  File "/home/name/Documents/eg/models.py", line 6, in <module>
    class Book:
  File "/home/name/Documents/eg/models.py", line 9, in Book
    def get_controller(self, some_bookcontroller:controllers.BookController):
AttributeError: partially initialized module 'controllers' has no attribute 'BookController' (most likely due to a circular import)

One can notice that Typeguard throws an error when a string is passed into BookController() instead of a Book(), meaning it correctly performs type checking.

Limitations

One should then think to design the code such that one does not need to call the models.py.

Answered By: a.t.

Your problem is that by using the denamespacing import form (from x import y) the contents of an imported module can’t be resolved lazily, so one side or the other will require a name before the other module has finished importing (and therefore before it has defined the name).

The typical solution here is to use namespaced imports (import x) and qualify your uses (x.y) so that, assuming the names aren’t needed to define your module, just when the functions within it are called, they can defer resolution to when they are needed and the circular aspect is not an issue.

Having checked the source code, typechecked is lazy and defers resolving annotations until the function in question is called, so it should be easy enough to fix this, changing your code to (inline comments indicating significant changes):

# models.py
import controllers  # Namespaced import

from typeguard import typechecked

class Book:
    @typechecked
    def get_controller(self, some_bookcontroller: controllers.BookController):  # Namespace qualified annotation
        return some_bookcontroller

And:

# controllers.py
from __future__ import annotations
from typing import TYPE_CHECKING
from typeguard import typechecked

import models  # Namespaced import

class BookController:
    @typechecked
    def __init__(self, book: models.Book) -> None:  # Namespace qualified annotation
        self.book = book

The code from models.py that does:

some_book=Book()
BookController("somestring")

should be moved from models to some other module that imports those names from the two modules with circular dependencies to make the code completely robust. This is because, being at top-level, depending on which of the cyclicly defined modules got imported first (the problem occurs with models.py here), it would be unable to resolve the name from controllers, even if the name is namespace-qualified (because it would be trying to use it before models finished being defined, and when the import of controllers pauses to wait on models to finish being defined, it hasn’t defined the rest of its contents, and therefore the code that runs on import in models can’t resolve the names from controllers).

If it’s in some other module importing from models and controllers, both of those imports will resolve by the time the subsequent code is run (assuming neither of them import this hypothetical module3 themselves), so it works either way (using namespaced or denamespaced imports).

If you’re curious how circular imports work in Python, there’s a complete explanation on What happens when using mutual or circular (cyclic) imports in Python?, but the short version is that, the very first time a module is imported in an entire program, Python puts an empty module object in the sys.modules cache, then runs the code within the module to populate the cached module object. The second time a module is imported, including when it’s imported by a module that imports the original module cyclically, Python just binds the cached module object and does nothing else, even if the cached module object isn’t populated yet.

This means that when an import in progress at the top of the stack (module B) tries to import a module that is already in progress below it on the stack (module A), module B gets the incompletely initialized module from the cache (because A was already in the process of being imported). If defining the contents of B relies on any component of A that was not defined before A tried to import B, it won’t exist on the cached (and mostly empty) A module. But so long as all such reference to A in B are confined to functions that aren’t called during definition time, or are used in annotations that from __future__ import annotations converts to string-based annotations and nothing tries to resolve them during the import process, this works fine. B finishes being defined without trying to use any elements from A, and when A‘s definition resumes, a complete B module exists for it to load from.

The problem occurs when the module in a cycle imported second (B in this case) tries to use a component of the module imported first (A) that isn’t defined until after A finishes importing B (usually pretty early on in A, so almost nothing in A is defined). from A import spam requires resolving A.spam immediately, so it’s going to break. import A, and using A.spam at top level (as you did with controllers.BookController in your original code) also breaks things.

Your use case was even nastier, because in fact, python models.py, there were secretly three modules involved:

  1. __main__ (which is the pseudoname given to the top level script, so the code is whatever is in models.py, but it is not the same thing as the models module for import/cache purposes), which imports…
  2. controllers, which in turn imports…
  3. models, which then cyclically imports controllers

This makes things uglier, because despite thinking you imported models first, you actually imported controllers before models, and the code in models.py is executed twice, once for the script itself as __main__, and once for the import as models. There are two unrelated Book classes (which can cause type-checking issues), and that top-level code in models.py was executed twice (creating two instances of BookController, and one instance each of the two unrelated but identical definitions of Book).

The solution is:

  1. Ensure both sides of the circular import use plain namespaced imports (import A/import B) and neither tries to access any component of the other at top-level (including at class definition level when the class is at top level), nor do they try to call any methods/functions/constructors from top-level that would indirectly access another component. Technically speaking, you could make a working use case without both of them being so careful, but it would be incredibly brittle; whichever module took an eager dependency on the other must be imported first, and in practice, you don’t want to assume that one module is always imported first (you could force it in a package structure that includes a __init__.py, but that breaks implicit namespace packages, and it’s still brittle from a maintainer’s point of view, even if it’s safe from an API consumer point of view), so it’s best to do the cautious thing on both sides.
  2. Never involve a module that will be invoked as a script (with python modulename.py or python -mmodulename) in a circular import scenario. Ideally never write a module that will be used as both script and imported module in the same run of the program (either it’s the main script and no one else imports it, or it’s imported as a module with some other script serving as main). If a module is both run as the top-level script and imported elsewhere, weird stuff happens.
Answered By: ShadowRanger