Automatically Update Python source code (imports)

You’ll need to write/find some script that will do text replacement of all occurrences in some folder. I remember that Notepad++ could do that.

But as you mentioned, that regexp will not help here, then no scripts (even Open Source) will help here too. And you’ll definitely need to have some intelligence here, that will build index of your dependencies/modules/files/packages/etc. and will be able to manipulate them on that level. And that is the purpose for which IDE were built for.

You can choose any that you like: PyCharm, Sublime, Visual Studio or any other that are not just text editor but something that has refactoring functionality.

In any case, I would suggest you to do following refactoring steps:

• rename old methods and their usages to new names
• then just replace package path & names in imports with newer versions

In cases where there is no obvious way to solve a batch editing problem, doing the next best thing with the addition of some manual work can work just as well.

As you’ve mentioned in your post:

Since above example is just one example of many ways a method can be imported, I don’t think a simple regular expression can help here.

I would recommend using a regular expression, while still printing out potential matches in case they are relevant:

def potential(line):
    # This is just a minimal example; replace with more reliable expression
    return "foo_method" in line or "a.b" in line 

matches = ["from a.b import foo_method"] # Add more to the list if necessary
new = "from b.d import bar_method" 
# new = "from b.d import bar_method as foo_method"

file = "file.py"
result = ""

with open(file) as f:
    for line in f:
        for match in matches:
            if match in line:
                result += line.replace(match, new)
                break
        else:
            if potential(line):
                print(line)

                # Here is the part where you manually check lines that potentially needs editing
                new_line = input("Replace with... (leave blank to ignore) ")
                if new_line:
                    result += new_line + "n"
                    continue
            result += line
                    
with open(file, "w") as f:
    f.write(result) 

Also, this goes without saying, but always make sure to create at least one backup of your original code base/project before doing such alterations.

But I don’t really think there will be too many complications on the different ways to import the method given that the code base was developed in proper PEP-8, as from What are all the ways to import modules in Python?:

The only ways that matter for ordinary usage are the first three ways listed on that page:

• import module
• from module import this, that, tother
• from module import *

Finally, to avoid complications in renaming each instance the files call the foo_method from foo_method to bar_method, I would recommend importing the newly named bar_method as foo_method, using the as keyword, of course.

A programmatic solution is to convert each file into a syntax tree, identify parts that meet your criteria and transform them. You can do this with Python’s ast module, but it doesn’t preserve whitespace or comments. There are also libraries that preserve these features because they operate on concrete (or lossless) rather than abstract syntax trees.

Red Baron is one such tool, but it does not support Python 3.8+, and looks to be unmaintained (last commit in 2019). libcst is another, and I’ll use it in this answer (disclaimer: I am not associated with the libcst project). Note that libcst does not yet support Python 3.10+.

The code below uses a Transformer that can identify

• from a.b import foo_method statements
• function calls where the function is named foo_method

and transforms the identified nodes into

• from b.d import bar_method
• bar_method

In the transformer class we specify methods named leave_Node where Node is the type of node that we want to inspect and transform (we can also specify visit_Node methods, but we don’t need them for this example). Within the methods we use matchers to check that the nodes match our criteria for transformation.

import libcst as cst
import libcst.matchers as m


src = """
import foo
from a.b import foo_method


class C:
    def do_something(self, x):
        return foo_method(x)
"""


class ImportFixer(cst.CSTTransformer):
    def leave_SimpleStatementLine(self, orignal_node, updated_node):
        """Replace imports that match our criteria."""
        if m.matches(updated_node.body[0], m.ImportFrom()):
            import_from = updated_node.body[0]
            if m.matches(
                import_from.module,
                m.Attribute(value=m.Name('a'), attr=m.Name('b')),
            ):
                if m.matches(
                    import_from.names[0],
                    m.ImportAlias(name=m.Name('foo_method')),
                ):
                    # Note that when matching we use m.Node,
                    # but when replacing we use cst.Node.
                    return updated_node.with_changes(
                        body=[
                            cst.ImportFrom(
                                module=cst.Attribute(
                                    value=cst.Name('b'), attr=cst.Name('d')
                                ),
                                names=[
                                    cst.ImportAlias(
                                        name=cst.Name('bar_method')
                                    )
                                ],
                            )
                        ]
                    )
        return updated_node

    def leave_Call(self, original_node, updated_node):
        if m.matches(updated_node, m.Call(func=m.Name('foo_method'))):
            return updated_node.with_changes(func=cst.Name('bar_method'))
        return updated_node


source_tree = cst.parse_module(src)
transformer = ImportFixer()
modified_tree = source_tree.visit(transformer)
print(modified_tree.code)

Output:

import foo
from b.d import bar_method


class C:
    def do_something(self, x):
        return bar_method(x)

You can use libcst‘s parsing helpers in the Python REPL to view and work with the node trees for modules, statements and expressions. This is usually the best way to work out which nodes to transform and what needs to be matched.

libcst provides a framework named codemods to support refactoring large codebases.