Recursive unittest discover

Question:

I have a package with a directory “tests” in which I’m storing my unit tests. My package looks like:

.
├── LICENSE
├── models
│   └── __init__.py
├── README.md
├── requirements.txt
├── tc.py
├── tests
│   ├── db
│   │   └── test_employee.py
│   └── test_tc.py
└── todo.txt

From my package directory, I want to be able to find both tests/test_tc.py and tests/db/test_employee.py. I’d prefer not to have to install a third-party library (nose or etc) or have to manually build a TestSuite to run this in.

Surely there’s a way to tell unittest discover not to stop looking once it’s found a test? python -m unittest discover -s tests will find tests/test_tc.py and python -m unittest discover -s tests/db will find tests/db/test_employee.py. Isn’t there a way to find both?

Asked By: Adam Smith

||
Source

Answers:

If you’re okay with adding a __init__.py file inside tests, you can put a load_tests function there that will handle discovery for you.

If a test package name (directory with __init__.py) matches the
pattern then the package will be checked for a ‘load_tests’ function. If
this exists then it will be called with loader, tests, pattern.

If load_tests exists then discovery does not recurse into the package,
load_tests is responsible for loading all tests in the package.

I’m far from confident that this is the best way, but one way to write that function would be:

import os
import pkgutil
import inspect
import unittest

# Add *all* subdirectories to this module's path
__path__ = [x[0] for x in os.walk(os.path.dirname(__file__))]

def load_tests(loader, suite, pattern):
    for imp, modname, _ in pkgutil.walk_packages(__path__):
        mod = imp.find_module(modname).load_module(modname)
        for memname, memobj in inspect.getmembers(mod):
            if inspect.isclass(memobj):
                if issubclass(memobj, unittest.TestCase):
                    print("Found TestCase: {}".format(memobj))
                    for test in loader.loadTestsFromTestCase(memobj):
                        print("  Found Test: {}".format(test))
                        suite.addTest(test)

    print("=" * 70)
    return suite

Pretty ugly, I agree.

First you add all subdirectories to the test packages’s path (Docs).

Then, you use pkgutil to walk the path, looking for packages or modules.

When it finds one, it then checks the module members to see whether they’re classes, and if they’re classes, whether they’re subclasses of unittest.TestCase. If they are, the tests inside the classes are loaded into the test suite.

So now, from inside your project root, you can type

python -m unittest discover -p tests

Using the -p pattern switch. If all goes well, you’ll see what I saw, which is something like:

Found TestCase: <class 'test_tc.TestCase'>
  Found Test: testBar (test_tc.TestCase)
  Found Test: testFoo (test_tc.TestCase)
Found TestCase: <class 'test_employee.TestCase'>
  Found Test: testBar (test_employee.TestCase)
  Found Test: testFoo (test_employee.TestCase)
======================================================================
....
----------------------------------------------------------------------
Ran 4 tests in 0.001s

OK

Which is what was expected, each of my two example files contained two tests, testFoo and testBar each.

Edit: After some more digging, it looks like you could specify this function as:

def load_tests(loader, suite, pattern):
    for imp, modname, _ in pkgutil.walk_packages(__path__):
        mod = imp.find_module(modname).load_module(modname)
        for test in loader.loadTestsFromModule(mod):
            print("Found Tests: {}".format(test._tests))
            suite.addTests(test)

This uses the loader.loadTestsFromModule() method instead of the loader.loadTestsFromTestCase() method I used above. It still modifies the tests package path and walks it looking for modules, which I think is the key here.

The output looks a bit different now, since we’re adding a found testsuite at a time to our main testsuite suite:

python -m unittest discover -p tests
Found Tests: [<test_tc.TestCase testMethod=testBar>, <test_tc.TestCase testMethod=testFoo>]
Found Tests: [<test_employee.TestCase testMethod=testBar>, <test_employee.TestCase testMethod=testFoo>]
======================================================================
....
----------------------------------------------------------------------
Ran 4 tests in 0.000s

OK

But we still get the 4 tests we expected, in both classes, in both subdirectories.

Answered By: jedwards

In doing a bit of digging, it seems that as long as deeper modules remain importable, they’ll be discovered via python -m unittest discover. The solution, then, was simply to add a __init__.py file to each directory to make them packages.

.
├── LICENSE
├── models
│   └── __init__.py
├── README.md
├── requirements.txt
├── tc.py
├── tests
│   ├── db
│   │   ├── __init__.py       # NEW
│   │   └── test_employee.py
│   ├── __init__.py           # NEW
│   └── test_tc.py
└── todo.txt

So long as each directory has an __init__.py, python -m unittest discover can import the relevant test_* module.

Answered By: Adam Smith

The point using init.py, is that one may encounters side effects, like file not being the script file path. Using FOR DOS command can help (not found of DOS commands, but sometimes it helps

setlocal
set CWD=%CD%
FOR /R %%T in (*_test.py) do (
  CD %%~pT
  python %%T
)
CD %CWD%
endlocal
  • /R allows for walkthrough the hierarchy from current folder.
  • (expr) allows for selecting test files (I use _test.py)
  • %%~pT is $(dirname $T) in shell.
  • I saved and restore my original directory, as the .bat leaves me where it ends
  • setlocal … endlocal to not pollute my environment with CWD.
Answered By: Xavier Fornari