Automatically generating Python type annotations?

Question

I’m retrofitting some old Python code to add PyType annotations. I’m doing this mechanically, by adding print statements to the top of each function

def f(a, b):
    print("type a:", type(a))
    print("type b:", type(b))

and updating accordingly.

Is there anything which can help automate this process?

Asked By: Mark Harrison

||

Source

Answer 1

Sure, there are some projects out there that can help out.

Instagram developed the MonkeyType tool to pick up runtime type information. These are output as stubs files, but you can use the tools included to apply those to your source files as inline annotations, or use the retype project to do the same.
Dropbox has a similar tool called pyannotate; this tool can generate either Python 2 type hint comments or annotations.

Neither project will produce perfect type hints; always take their output as a starting point, not the authoritative final annotations. Quoting the MonkeyType readme:

With MonkeyType, it’s very easy to add annotations that reflect the concrete types you use at runtime, but those annotations may not always match the full intended capability of the functions. For instance, add is capable of handling many more types than just integers. Similarly, MonkeyType may generate a concrete List annotation where an abstract Sequence or Iterable would be more appropriate. MonkeyType’s annotations are an informative first draft, to be checked and corrected by a developer.

But do read the Instagram engineering blog post on MonkeyType; it has been instrumental in adding type annotations to the Instagram codebase.

Disclaimer: MonkeyType and retype are both projects by colleagues at Facebook. I have not had any input in those tools myself.

Answered By: Martijn Pieters

Answer 2

I found the given answer to be a bit abstract, so here is a concrete (copy-paste) solution I verified on a project with the following structure:

projectname/
|-- src/
|   |-- projectname/__main__.py
|   |-- projectname/some_folder/some_script.py
|-- tests/
|   |-- projectname/test_something.py
|   |-- projectname/another_folder/test_another_thing.py
|
|-- setup.py
|-- README.md

The main code is executed with:

python -m src.projectname

and the tests are executed with:

python -m pytest

Description

This solution uses the tests one has in tests to generate the stubs. It uses the pyannotate project by Dropbox.

Limitations

It only generates stubs for functions that are used/called by a test.
I did not yet find out how to also generate the type-hints for the code called by src/projectname/__main__.py, however that also seems possible.

1. Generate Type hints

Generate a .py file that takes over running the tests, to ensure the type-hints are generated.

Copy paste:

# Configuration for pytest to automatically collect types.
# Thanks to Guilherme Salgado.

import pytest


def pytest_collection_finish(session):
    """Handle the pytest collection finish hook: configure pyannotate.

    Explicitly delay importing `collect_types` until all tests have
    been collected.  This gives gevent a chance to monkey patch the
    world before importing pyannotate.
    """
    from pyannotate_runtime import collect_types
    collect_types.init_types_collection()


@pytest.fixture(autouse=True)
def collect_types_fixture():
    from pyannotate_runtime import collect_types
    collect_types.start()
    yield
    collect_types.stop()


def pytest_sessionfinish(session, exitstatus):
    from pyannotate_runtime import collect_types
    collect_types.dump_stats("type_info.json")

into tests/conftest.py.

2. Generate the type-hints.

Run:

pip install pyannotate
python -m pytest

This should output a file named: type_info.json in the root directory of the project. That file contains the type-hints.

3. Apply the type hints to your code

Run:

for f in $(find src/ -name '*.py'); do pyannotate -w --python-version 3 --type-info type_info.json $f; done
for f in $(find tests/ -name '*.py'); do pyannotate -w --python-version 3 --type-info type_info.json $f; done

That should apply the type hints to the code. You can verify the changes with:

git diff

Answered By: a.t.