Unit test script returns exit code = 0 even if tests fail
Question:
My testing script looks as follows:
import os
import sys
from unittest import defaultTestLoader as loader, TextTestRunner
path_to_my_project = os.path.dirname(os.path.abspath(__file__)) + '/../'
sys.path.insert(0, path_to_my_project)
suite = loader.discover('my_project')
runner = TextTestRunner()
runner.run(suite)
If I run this script, the output is:
$ python3 runtest.py
.....F.....
======================================================================
FAIL: test_insert (fate.test.test_operators.OperatorTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/chiel/Projects/tfate/libs/fate/../fate/test/test_operators.py", line 16, in test_insert
self.assertEqual(expected, self.session.text[:14])
AssertionError: 'Foo import sys$' != 'Foo import sys'
- Foo import sys$
? -
+ Foo import sys
----------------------------------------------------------------------
Ran 12 tests in 0.030s
FAILED (failures=1)
And exit code zero:
$ echo $?
0
However, the Python documentation states that "By default main calls sys.exit()
with an exit code indicating success or failure of the tests run."
What is wrong with my script?
Answers:
The code is not using unittest.main
. You need to check the result using TestResult.wasSuccessful
and call sys.exit
manually.
import sys
....
ret = not runner.run(suite).wasSuccessful()
sys.exit(ret)
I had some trouble getting TextTestRunner
results. For those like me, here is how it works:
"""Run all tests inside of *_test.py modules located in the same directory."""
import sys
import unittest
if __name__ == '__main__':
test_suite = unittest.defaultTestLoader.discover('.', '*_test.py')
test_runner = unittest.TextTestRunner(resultclass=unittest.TextTestResult)
result = test_runner.run(test_suite)
sys.exit(not result.wasSuccessful())
My testing script looks as follows:
import os
import sys
from unittest import defaultTestLoader as loader, TextTestRunner
path_to_my_project = os.path.dirname(os.path.abspath(__file__)) + '/../'
sys.path.insert(0, path_to_my_project)
suite = loader.discover('my_project')
runner = TextTestRunner()
runner.run(suite)
If I run this script, the output is:
$ python3 runtest.py
.....F.....
======================================================================
FAIL: test_insert (fate.test.test_operators.OperatorTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/chiel/Projects/tfate/libs/fate/../fate/test/test_operators.py", line 16, in test_insert
self.assertEqual(expected, self.session.text[:14])
AssertionError: 'Foo import sys$' != 'Foo import sys'
- Foo import sys$
? -
+ Foo import sys
----------------------------------------------------------------------
Ran 12 tests in 0.030s
FAILED (failures=1)
And exit code zero:
$ echo $?
0
However, the Python documentation states that "By default main calls sys.exit()
with an exit code indicating success or failure of the tests run."
What is wrong with my script?
The code is not using unittest.main
. You need to check the result using TestResult.wasSuccessful
and call sys.exit
manually.
import sys
....
ret = not runner.run(suite).wasSuccessful()
sys.exit(ret)
I had some trouble getting TextTestRunner
results. For those like me, here is how it works:
"""Run all tests inside of *_test.py modules located in the same directory."""
import sys
import unittest
if __name__ == '__main__':
test_suite = unittest.defaultTestLoader.discover('.', '*_test.py')
test_runner = unittest.TextTestRunner(resultclass=unittest.TextTestResult)
result = test_runner.run(test_suite)
sys.exit(not result.wasSuccessful())