Python library 'unittest': Generate multiple tests programmatically

nose (suggested by @Paul Hankin)

#!/usr/bin/env python
# file: test_pairs_nose.py
from nose.tools import eq_ as eq

from mymodule import f

def test_pairs():
    for input, output in [ (2, 332), (234, 99213), (9, 3), ]:
        yield _test_f, input, output

def _test_f(input, output):
    try:
        eq(f(input), output)
    except AssertionError:
        if input == 9: # expected failure
            from nose.exc import SkipTest
            raise SkipTest("expected failure")
        else:
            raise

if __name__=="__main__":
   import nose; nose.main()

Example:

$ nosetests test_pairs_nose -v
test_pairs_nose.test_pairs(2, 332) ... ok
test_pairs_nose.test_pairs(234, 99213) ... ok
test_pairs_nose.test_pairs(9, 3) ... SKIP: expected failure

----------------------------------------------------------------------
Ran 3 tests in 0.001s

OK (SKIP=1)

unittest (an approach similar to @doublep’s one)

#!/usr/bin/env python
import unittest2 as unittest
from mymodule import f

def add_tests(generator):
    def class_decorator(cls):
        """Add tests to `cls` generated by `generator()`."""
        for f, input, output in generator():
            test = lambda self, i=input, o=output, f=f: f(self, i, o)
            test.__name__ = "test_%s(%r, %r)" % (f.__name__, input, output)
            setattr(cls, test.__name__, test)
        return cls
    return class_decorator

def _test_pairs():
    def t(self, input, output):
        self.assertEqual(f(input), output)

    for input, output in [ (2, 332), (234, 99213), (9, 3), ]:
        tt = t if input != 9 else unittest.expectedFailure(t)
        yield tt, input, output

class TestCase(unittest.TestCase):
    pass
TestCase = add_tests(_test_pairs)(TestCase)

if __name__=="__main__":
    unittest.main()

Example:

$ python test_pairs_unit2.py -v
test_t(2, 332) (__main__.TestCase) ... ok
test_t(234, 99213) (__main__.TestCase) ... ok
test_t(9, 3) (__main__.TestCase) ... expected failure

----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK (expected failures=1)

If you don’t want to install unittest2 then add:

try:
    import unittest2 as unittest
except ImportError:
    import unittest
    if not hasattr(unittest, 'expectedFailure'):
       import functools
       def _expectedFailure(func):
           @functools.wraps(func)
           def wrapper(*args, **kwargs):
               try:
                   func(*args, **kwargs)
               except AssertionError:
                   pass
               else:
                   raise AssertionError("UnexpectedSuccess")
           return wrapper
       unittest.expectedFailure = _expectedFailure

I had to do something similar. I created simple TestCase subclasses that took a value in their __init__, like this:

class KnownGood(unittest.TestCase):
    def __init__(self, input, output):
        super(KnownGood, self).__init__()
        self.input = input
        self.output = output
    def runTest(self):
        self.assertEqual(function_to_test(self.input), self.output)

I then made a test suite with these values:

def suite():
    suite = unittest.TestSuite()
    suite.addTests(KnownGood(input, output) for input, output in known_values)
    return suite

You can then run it from your main method:

if __name__ == '__main__':
    unittest.TextTestRunner().run(suite())

The advantages of this are:

• As you add more values, the number of reported tests increases, which makes you feel like you are doing more.
• Each individual test case can fail individually
• It’s conceptually simple, since each input/output value is converted into one TestCase

As often with Python, there is a complicated way to provide a simple solution.

In that case, we can use metaprogramming, decorators, and various nifty Python tricks to achieve a nice result. Here is what the final test will look like:

import unittest

# Some magic code will be added here later

class DummyTest(unittest.TestCase):
  @for_examples(1, 2)
  @for_examples(3, 4)
  def test_is_smaller_than_four(self, value):
    self.assertTrue(value < 4)

  @for_examples((1,2),(2,4),(3,7))
  def test_double_of_X_is_Y(self, x, y):
    self.assertEqual(2 * x, y)

if __name__ == "__main__":
  unittest.main()

When executing this script, the result is:

..F...F
======================================================================
FAIL: test_double_of_X_is_Y(3,7)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/xdecoret/Documents/foo.py", line 22, in method_for_example
    method(self, *example)
  File "/Users/xdecoret/Documents/foo.py", line 41, in test_double_of_X_is_Y
    self.assertEqual(2 * x, y)
AssertionError: 6 != 7

======================================================================
FAIL: test_is_smaller_than_four(4)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/xdecoret/Documents/foo.py", line 22, in method_for_example
    method(self, *example)
  File "/Users/xdecoret/Documents/foo.py", line 37, in test_is_smaller_than_four
    self.assertTrue(value < 4)
AssertionError

----------------------------------------------------------------------
Ran 7 tests in 0.001s

FAILED (failures=2)

which achieves our goal:

• it is unobtrusive: we derive from TestCase as usual
• we write parametrized tests only once
• each example value is considered an individual test
• the decorator can be stacked, so it is easy to use sets of examples (e.g., using a function to build the list of values from example files or directories)
• The icing on the cake is it works for arbitrary arity of the signature

So how does it work? Basically, the decorator stores the examples in an attribute of the function. We use a metaclass to replace every decorated function with a list of functions. And we replace the unittest.TestCase with our new magic code (to be pasted in the "magic" comment above) is:

__examples__ = "__examples__"

def for_examples(*examples):
    def decorator(f, examples=examples):
      setattr(f, __examples__, getattr(f, __examples__,()) + examples)
      return f
    return decorator

class TestCaseWithExamplesMetaclass(type):
  def __new__(meta, name, bases, dict):
    def tuplify(x):
      if not isinstance(x, tuple):
        return (x,)
      return x
    for methodname, method in dict.items():
      if hasattr(method, __examples__):
        dict.pop(methodname)
        examples = getattr(method, __examples__)
        delattr(method, __examples__)
        for example in (tuplify(x) for x in examples):
          def method_for_example(self, method = method, example = example):
            method(self, *example)
          methodname_for_example = methodname + "(" + ", ".join(str(v) for v in example) + ")"
          dict[methodname_for_example] = method_for_example
    return type.__new__(meta, name, bases, dict)

class TestCaseWithExamples(unittest.TestCase):
  __metaclass__ = TestCaseWithExamplesMetaclass
  pass

unittest.TestCase = TestCaseWithExamples

If someone wants to package this nicely, or propose a patch for unittest, feel free! A quote of my name will be appreciated.

The code can be made much simpler and fully encapsulated in the decorator if you are ready to use frame introspection (import the sys module)

def for_examples(*parameters):

  def tuplify(x):
    if not isinstance(x, tuple):
      return (x,)
    return x

  def decorator(method, parameters=parameters):
    for parameter in (tuplify(x) for x in parameters):

      def method_for_parameter(self, method=method, parameter=parameter):
        method(self, *parameter)
      args_for_parameter = ",".join(repr(v) for v in parameter)
      name_for_parameter = method.__name__ + "(" + args_for_parameter + ")"
      frame = sys._getframe(1)  # pylint: disable-msg=W0212
      frame.f_locals[name_for_parameter] = method_for_parameter
    return None
  return decorator

I think Rory’s solution is the cleanest and shortest. However, this variation of doublep’s "create synthetic functions in a TestCase" also works:

from functools import partial

class TestAllReports(unittest.TestCase):
    pass

def test_spamreport(name):
    assert classify(getSample(name))=='spamreport', name

for rep in REPORTS:
    testname = 'test_' + rep
    testfunc = partial(test_spamreport, rep)
    testfunc.__doc__ = testname
    setattr(TestAllReports, testname, testfunc)

if __name__=='__main__':
    unittest.main(argv=sys.argv + ['--verbose'])

Some of the tools available for doing parametrized tests in Python are:

• Nose test generators (only for function tests, not TestCase classes)
• nose-parametrized by David Wolever (also for TestCase classes)
• Unittest template by Boris Feld
• Parametrized tests in py.test
• parametrized-testcase by Austin Bingham

See also question 1676269 for more answers to this question.