changing order of unit tests in Python
Question:
How can I make it so unit tests in Python (using unittest
) are run in the order in which they are specified in the file?
Answers:
Clever Naming.
class Test01_Run_Me_First( unittest.TestCase ):
def test010_do_this( self ):
assertTrue( True )
def test020_do_that( self ):
etc.
Is one way to force a specific order.
There are also test runners which do that by themselves – I think py.test does it.
You can change the default sorting behavior by setting a custom comparison function. In unittest.py you can find the class variable unittest.TestLoader.sortTestMethodsUsing
which is set to the builtin function cmp
by default.
For example you can revert the execution order of your tests with doing this:
import unittest
unittest.TestLoader.sortTestMethodsUsing = lambda _, x, y: cmp(y, x)
Use proboscis library as I mentioned already (please see short description there).
As said above, normally tests in test cases should be tested in any (i.e. random) order.
However, if you do want to order the tests in the test case, apparently it is not trivial.
Tests (method names) are retrieved from test cases using dir(MyTest)
, which returns a sorted list of members. You can use a clever (?) hack to order methods by their line numbers. This will work for one test case:
if __name__ == "__main__":
loader = unittest.TestLoader()
ln = lambda f: getattr(MyTestCase, f).im_func.func_code.co_firstlineno
lncmp = lambda a, b: cmp(ln(a), ln(b))
loader.sortTestMethodsUsing = lncmp
unittest.main(testLoader=loader, verbosity=2)
I found a solution for it using PyTest ordering plugin provided here.
Try py.test YourModuleName.py -vv
in CLI and the test will run in the order they have appeared in your module.
I did the same thing and works fine for me.
Note: You need to install PyTest package and import it.
hacky way (run this file in pycharm or other unit test runner)
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
import unittest
def make_suite():
class Test(unittest.TestCase):
def test_32(self):
print "32"
def test_23(self):
print "23"
suite = unittest.TestSuite()
suite.addTest(Test('test_32'))
suite.addTest(Test('test_23'))
return suite
suite = make_suite()
class T(unittest.TestCase):
counter = 0
def __call__(self, *args, **kwargs):
res = suite._tests[T.counter](*args, **kwargs)
T.counter += 1
return res
for t in suite._tests:
name = "{}${}".format(t._testMethodName, t.__class__.__name__)
setattr(T, name, t)
The default order is alphabetical. You can put test_<int>
before every test to specify the order of execution.
Also you can set unittest.TestLoader.sortTestMethodsUsing = None
to eliminate the sort.
Check out Unittest Documentation for more details.
How can I make it so unit tests in Python (using unittest
) are run in the order in which they are specified in the file?
Clever Naming.
class Test01_Run_Me_First( unittest.TestCase ):
def test010_do_this( self ):
assertTrue( True )
def test020_do_that( self ):
etc.
Is one way to force a specific order.
There are also test runners which do that by themselves – I think py.test does it.
You can change the default sorting behavior by setting a custom comparison function. In unittest.py you can find the class variable unittest.TestLoader.sortTestMethodsUsing
which is set to the builtin function cmp
by default.
For example you can revert the execution order of your tests with doing this:
import unittest
unittest.TestLoader.sortTestMethodsUsing = lambda _, x, y: cmp(y, x)
Use proboscis library as I mentioned already (please see short description there).
As said above, normally tests in test cases should be tested in any (i.e. random) order.
However, if you do want to order the tests in the test case, apparently it is not trivial.
Tests (method names) are retrieved from test cases using dir(MyTest)
, which returns a sorted list of members. You can use a clever (?) hack to order methods by their line numbers. This will work for one test case:
if __name__ == "__main__":
loader = unittest.TestLoader()
ln = lambda f: getattr(MyTestCase, f).im_func.func_code.co_firstlineno
lncmp = lambda a, b: cmp(ln(a), ln(b))
loader.sortTestMethodsUsing = lncmp
unittest.main(testLoader=loader, verbosity=2)
I found a solution for it using PyTest ordering plugin provided here.
Try py.test YourModuleName.py -vv
in CLI and the test will run in the order they have appeared in your module.
I did the same thing and works fine for me.
Note: You need to install PyTest package and import it.
hacky way (run this file in pycharm or other unit test runner)
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
import unittest
def make_suite():
class Test(unittest.TestCase):
def test_32(self):
print "32"
def test_23(self):
print "23"
suite = unittest.TestSuite()
suite.addTest(Test('test_32'))
suite.addTest(Test('test_23'))
return suite
suite = make_suite()
class T(unittest.TestCase):
counter = 0
def __call__(self, *args, **kwargs):
res = suite._tests[T.counter](*args, **kwargs)
T.counter += 1
return res
for t in suite._tests:
name = "{}${}".format(t._testMethodName, t.__class__.__name__)
setattr(T, name, t)
The default order is alphabetical. You can put test_<int>
before every test to specify the order of execution.
Also you can set unittest.TestLoader.sortTestMethodsUsing = None
to eliminate the sort.
Check out Unittest Documentation for more details.