How to show the error messages caught by assertRaises() in unittest in Python2.7?
Question:
In order to make sure that the error messages from my module are informative, I would like to see all the error messages caught by assertRaises(). Today I do it for each assertRaises(), but as there are lots of them in the test code it gets very tedious.
How can I print the error messages for all the assertRaises()? I have studied the documentation on http://docs.python.org/library/unittest.html without figuring out how to solve it. Can I somehow monkeypatch the assertRaises() method? I prefer not to change all the assertRaises() lines in the test code, as I most often use the test code the standard way.
I guess this question is related to Python unittest: how do I test the argument in an Exceptions?
This is how I do it today. For example:
#!/usr/bin/env python
def fail():
raise ValueError('Misspellled errrorr messageee')
And the test code:
#!/usr/bin/env python
import unittest
import failure
class TestFailureModule(unittest.TestCase):
def testFail(self):
self.assertRaises(ValueError, failure.fail)
if __name__ == '__main__':
unittest.main()
To check the error message, I simply change the error type in the assertRaises() to for example IOError. Then I can see the error message:
E
======================================================================
ERROR: testFail (__main__.TestFailureModule)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_failure.py", line 8, in testFail
self.assertRaises(IOError, failure.fail)
File "/usr/lib/python2.7/unittest/case.py", line 471, in assertRaises
callableObj(*args, **kwargs)
File "/home/jonas/Skrivbord/failure.py", line 4, in fail
raise ValueError('Misspellled errrorr messageee')
ValueError: Misspellled errrorr messageee
----------------------------------------------------------------------
Ran 1 test in 0.001s
FAILED (errors=1)
Any suggestions?
/Jonas
EDIT:
With the hints from Robert Rossney I managed to solve the problem. It is not mainly intended for spelling errors, but for making sure that the error messages are really meaningful for the user of the module. The normal functionality of unittest (this is how I use it most of the time) is achieved by setting SHOW_ERROR_MESSAGES = False.
I simply override the assertRaises() method, as seen below. It works like charm!
SHOW_ERROR_MESSAGES = True
class NonexistantError(Exception):
pass
class ExtendedTestCase(unittest.TestCase):
def assertRaises(self, excClass, callableObj, *args, **kwargs):
if SHOW_ERROR_MESSAGES:
excClass = NonexistantError
try:
unittest.TestCase.assertRaises(self, excClass, callableObj, *args, **kwargs)
except:
print 'n ' + repr(sys.exc_info()[1])
A fraction of the resulting output:
testNotIntegerInput (__main__.TestCheckRegisteraddress) ...
TypeError('The registeraddress must be an integer. Given: 1.0',)
TypeError("The registeraddress must be an integer. Given: '1'",)
TypeError('The registeraddress must be an integer. Given: [1]',)
TypeError('The registeraddress must be an integer. Given: None',)
ok
testCorrectNumberOfBytes (__main__.TestCheckResponseNumberOfBytes) ... ok
testInconsistentLimits (__main__.TestCheckNumerical) ...
ValueError('The maxvalue must not be smaller than minvalue. Given: 45 and 47, respectively.',)
ValueError('The maxvalue must not be smaller than minvalue. Given: 45.0 and 47.0, respectively.',)
ok
testWrongValues (__main__.TestCheckRegisteraddress) ...
ValueError('The registeraddress is too small: -1, but minimum value is 0.',)
ValueError('The registeraddress is too large: 65536, but maximum value is 65535.',)
ok
testTooShortString (__main__.TestCheckResponseWriteData) ...
ValueError("The payload is too short: 2, but minimum value is 4. Given: '\x00X'",)
ValueError("The payload is too short: 0, but minimum value is 4. Given: ''",)
ValueError("The writedata is too short: 1, but minimum value is 2. Given: 'X'",)
ValueError("The writedata is too short: 0, but minimum value is 2. Given: ''",)
ok
testKnownValues (__main__.TestCreateBitPattern) ... ok
testNotIntegerInput (__main__.TestCheckSlaveaddress) ...
TypeError('The slaveaddress must be an integer. Given: 1.0',)
TypeError("The slaveaddress must be an integer. Given: '1'",)
TypeError('The slaveaddress must be an integer. Given: [1]',)
TypeError('The slaveaddress must be an integer. Given: None',)
ok
Answers:
Out-of-the-box unittest doesn’t do this. If this is something you want to do frequently, you can try something like this:
class ExtendedTestCase(unittest.TestCase):
def assertRaisesWithMessage(self, msg, func, *args, **kwargs):
try:
func(*args, **kwargs)
self.assertFail()
except Exception as inst:
self.assertEqual(inst.message, msg)
Derive your unit test classes from ExtendedTestCase instead of unittest.TestCase.
But really, if you’re simply concerned about misspelled error messages, and concerned enough to want to build test cases around it, you shouldn’t be inlining messages as string literals. You should do with them what you do with any other important strings: defining them as constants in a module that you import and that someone is responsible for proofreading. A developer who misspells words in his code will also misspell them in his test cases.
I once preferred the most excellent answer given above by @Robert Rossney. Nowadays, I prefer to use assertRaises as a context manager (a new capability in unittest2) like so:
with self.assertRaises(TypeError) as cm:
failure.fail()
self.assertEqual(
'The registeraddress must be an integer. Given: 1.0',
str(cm.exception)
)
You are looking for assertRaisesRegex, which is available since Python 3.2. From the docs:
self.assertRaisesRegex(ValueError, "invalid literal for.*XYZ'$",
int, 'XYZ')
or:
with self.assertRaisesRegex(ValueError, 'literal'):
int('XYZ')
PS: if you are using Python 2.7, then the correct method name is assertRaisesRegexp.
mkelley33 gives nice answer, but this approach can be detected as issue by some code analysis tools like Codacy. The problem is that it doesn’t know that assertRaises can be used as context manager and it reports that not all arguments are passed to assertRaises method.
So, I’d like to improve Robert’s Rossney answer:
class TestCaseMixin(object):
def assertRaisesWithMessage(self, exception_type, message, func, *args, **kwargs):
try:
func(*args, **kwargs)
except exception_type as e:
self.assertEqual(e.args[0], message)
else:
self.fail('"{0}" was expected to throw "{1}" exception'
.format(func.__name__, exception_type.__name__))
Key differences are:
- We test type of exception.
- We can run this code both on Python 2 and
Python 3 (we call e.args[0] because errors in Py3 don’t have
message attribute).
If you want the error message exactly match something:
with self.assertRaises(ValueError) as error:
do_something()
self.assertEqual(error.exception.message, 'error message')
In order to make sure that the error messages from my module are informative, I would like to see all the error messages caught by assertRaises(). Today I do it for each assertRaises(), but as there are lots of them in the test code it gets very tedious.
How can I print the error messages for all the assertRaises()? I have studied the documentation on http://docs.python.org/library/unittest.html without figuring out how to solve it. Can I somehow monkeypatch the assertRaises() method? I prefer not to change all the assertRaises() lines in the test code, as I most often use the test code the standard way.
I guess this question is related to Python unittest: how do I test the argument in an Exceptions?
This is how I do it today. For example:
#!/usr/bin/env python
def fail():
raise ValueError('Misspellled errrorr messageee')
And the test code:
#!/usr/bin/env python
import unittest
import failure
class TestFailureModule(unittest.TestCase):
def testFail(self):
self.assertRaises(ValueError, failure.fail)
if __name__ == '__main__':
unittest.main()
To check the error message, I simply change the error type in the assertRaises() to for example IOError. Then I can see the error message:
E
======================================================================
ERROR: testFail (__main__.TestFailureModule)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_failure.py", line 8, in testFail
self.assertRaises(IOError, failure.fail)
File "/usr/lib/python2.7/unittest/case.py", line 471, in assertRaises
callableObj(*args, **kwargs)
File "/home/jonas/Skrivbord/failure.py", line 4, in fail
raise ValueError('Misspellled errrorr messageee')
ValueError: Misspellled errrorr messageee
----------------------------------------------------------------------
Ran 1 test in 0.001s
FAILED (errors=1)
Any suggestions?
/Jonas
EDIT:
With the hints from Robert Rossney I managed to solve the problem. It is not mainly intended for spelling errors, but for making sure that the error messages are really meaningful for the user of the module. The normal functionality of unittest (this is how I use it most of the time) is achieved by setting SHOW_ERROR_MESSAGES = False.
I simply override the assertRaises() method, as seen below. It works like charm!
SHOW_ERROR_MESSAGES = True
class NonexistantError(Exception):
pass
class ExtendedTestCase(unittest.TestCase):
def assertRaises(self, excClass, callableObj, *args, **kwargs):
if SHOW_ERROR_MESSAGES:
excClass = NonexistantError
try:
unittest.TestCase.assertRaises(self, excClass, callableObj, *args, **kwargs)
except:
print 'n ' + repr(sys.exc_info()[1])
A fraction of the resulting output:
testNotIntegerInput (__main__.TestCheckRegisteraddress) ...
TypeError('The registeraddress must be an integer. Given: 1.0',)
TypeError("The registeraddress must be an integer. Given: '1'",)
TypeError('The registeraddress must be an integer. Given: [1]',)
TypeError('The registeraddress must be an integer. Given: None',)
ok
testCorrectNumberOfBytes (__main__.TestCheckResponseNumberOfBytes) ... ok
testInconsistentLimits (__main__.TestCheckNumerical) ...
ValueError('The maxvalue must not be smaller than minvalue. Given: 45 and 47, respectively.',)
ValueError('The maxvalue must not be smaller than minvalue. Given: 45.0 and 47.0, respectively.',)
ok
testWrongValues (__main__.TestCheckRegisteraddress) ...
ValueError('The registeraddress is too small: -1, but minimum value is 0.',)
ValueError('The registeraddress is too large: 65536, but maximum value is 65535.',)
ok
testTooShortString (__main__.TestCheckResponseWriteData) ...
ValueError("The payload is too short: 2, but minimum value is 4. Given: '\x00X'",)
ValueError("The payload is too short: 0, but minimum value is 4. Given: ''",)
ValueError("The writedata is too short: 1, but minimum value is 2. Given: 'X'",)
ValueError("The writedata is too short: 0, but minimum value is 2. Given: ''",)
ok
testKnownValues (__main__.TestCreateBitPattern) ... ok
testNotIntegerInput (__main__.TestCheckSlaveaddress) ...
TypeError('The slaveaddress must be an integer. Given: 1.0',)
TypeError("The slaveaddress must be an integer. Given: '1'",)
TypeError('The slaveaddress must be an integer. Given: [1]',)
TypeError('The slaveaddress must be an integer. Given: None',)
ok
Out-of-the-box unittest doesn’t do this. If this is something you want to do frequently, you can try something like this:
class ExtendedTestCase(unittest.TestCase):
def assertRaisesWithMessage(self, msg, func, *args, **kwargs):
try:
func(*args, **kwargs)
self.assertFail()
except Exception as inst:
self.assertEqual(inst.message, msg)
Derive your unit test classes from ExtendedTestCase instead of unittest.TestCase.
But really, if you’re simply concerned about misspelled error messages, and concerned enough to want to build test cases around it, you shouldn’t be inlining messages as string literals. You should do with them what you do with any other important strings: defining them as constants in a module that you import and that someone is responsible for proofreading. A developer who misspells words in his code will also misspell them in his test cases.
I once preferred the most excellent answer given above by @Robert Rossney. Nowadays, I prefer to use assertRaises as a context manager (a new capability in unittest2) like so:
with self.assertRaises(TypeError) as cm:
failure.fail()
self.assertEqual(
'The registeraddress must be an integer. Given: 1.0',
str(cm.exception)
)
You are looking for assertRaisesRegex, which is available since Python 3.2. From the docs:
self.assertRaisesRegex(ValueError, "invalid literal for.*XYZ'$",
int, 'XYZ')
or:
with self.assertRaisesRegex(ValueError, 'literal'):
int('XYZ')
PS: if you are using Python 2.7, then the correct method name is assertRaisesRegexp.
mkelley33 gives nice answer, but this approach can be detected as issue by some code analysis tools like Codacy. The problem is that it doesn’t know that assertRaises can be used as context manager and it reports that not all arguments are passed to assertRaises method.
So, I’d like to improve Robert’s Rossney answer:
class TestCaseMixin(object):
def assertRaisesWithMessage(self, exception_type, message, func, *args, **kwargs):
try:
func(*args, **kwargs)
except exception_type as e:
self.assertEqual(e.args[0], message)
else:
self.fail('"{0}" was expected to throw "{1}" exception'
.format(func.__name__, exception_type.__name__))
Key differences are:
- We test type of exception.
- We can run this code both on Python 2 and
Python 3 (we calle.args[0]because errors in Py3 don’t have
messageattribute).
If you want the error message exactly match something:
with self.assertRaises(ValueError) as error:
do_something()
self.assertEqual(error.exception.message, 'error message')