How should we test exceptions with nose?
Question:
I’m testing exceptions with nose. Here’s an example:
def testDeleteUserUserNotFound(self):
"Test exception is raised when trying to delete non-existent users"
try:
self.client.deleteUser('10000001-0000-0000-1000-100000000000')
# make nose fail here
except UserNotFoundException:
assert True
The assert is executed if the exception is raised, but if no exception is raised, it won’t be executed.
Is there anything I can put on the commented line above so that if no exception is raised nose will report a failure?
Answers:
I don’t know what nose is, but have you tried using ‘else’ after the except clause. I.e.
else:
assert False
def testDeleteUserUserNotFound(self):
"Test exception is raised when trying to delete non-existent users"
try:
self.client.deleteUser('10000001-0000-0000-1000-100000000000')
assert False # <---
except UserNotFoundException:
assert True
The semantics of try/except imply that the flow of execution leaves the try block on an exception, so assert False will not run if an exception is raised. Also, execution will not re-enter the try block again after the except block is done running, so you shouldn’t run into trouble.
↓
(statements)
↓ exception
(try) ↚──────────→ (except)
↓ │
(statements) ←───────────┘
↓
nose provides tools for testing exceptions (like unittest does).
Try this example (and read about the other tools at Nose Testing Tools
from nose.tools import *
l = []
d = dict()
@raises(Exception)
def test_Exception1():
'''this test should pass'''
l.pop()
@raises(KeyError)
def test_Exception2():
'''this test should pass'''
d[1]
@raises(KeyError)
def test_Exception3():
'''this test should fail (IndexError raised but KeyError was expected)'''
l.pop()
def test_Exception4():
'''this test should fail with KeyError'''
d[1]
I would think that this is the proper way that you were looking for because it lets you be specific about the exceptions that you expect or want. So you actually provoke the error to see that it raises the right exception. And then you let nose evaluate the result. (Put as little logic into the unit tests as possible!)
I do not know why it is not here yet but exist one more way:
import unittest
class TestCase(unittest.TestCase):
def testKeyError(self):
d = dict()
with self.assertRaises(KeyError):
d[1]
Use assert_raises:
from nose.tools import assert_raises
our_method = self.client.deleteUser
arg1 = '10000001-0000-0000-1000-100000000000'
expected_exception = UserNotFoundException
assert_raises(expected_exception, our_method, arg1)
Using try and catch in your tests seems like bad practice (most cases).
There’s no specific documentation in nose because it’s basically just a wrapper around unittest.TestCase.assertRaises (ref. How to use nose's assert_raises?)
I strongly recommend using assert_raises and assert_raises_regexp from nose.tools, which duplicate the behavior of assertRaises and assertRaisesRegexp from unittest.TestCase. These allow using the same functionality as provided by unittest.TestCase in test suites that do not actually use the unittest.TestCase class.
I find that @raises is much too blunt an instrument. Here is code illustrating the problem:
from nose.tools import *
something = ["aaa", "bbb"]
def foo(x, source=None):
if source is None:
source = something
return source[x]
# This is fine
@raises(IndexError)
def test1():
foo(3)
# This is fine. The expected error does not happen because we made
# a mistake in the test or in the code. The failure indicates we made
# a mistake.
@raises(IndexError)
def test2():
foo(1)
# This passes for the wrong reasons.
@raises(IndexError)
def test3():
source = something[2] # This is the line that raises the exception.
foo(10, source) # This is not tested.
# When we use assert_raises, we can isolate the line where we expect
# the failure. This causes an error due to the exception raised in
# the first line of the function.
def test4():
source = something[2]
with assert_raises(IndexError):
foo(10, source)
test3 passes, but not because foo has raised the exception we were expecting but because the code that sets up the data to be used by foo fails with the same exception. test4 shows how the test can be written using assert_raises to actually test what we mean to be testing. The problem on the first line will cause Nose to report an error and then we can rewrite the test so that that line so that we can finally test what we did mean to test.
@raises does not allow testing the message associated with the exception. When I raise ValueError, just to take one example, I usually want to raise it with an informative message. Here’s an example:
def bar(arg):
if arg: # This is incorrect code.
raise ValueError("arg should be higher than 3")
if arg >= 10:
raise ValueError("arg should be less than 10")
# We don't know which of the possible `raise` statements was reached.
@raises(ValueError)
def test5():
bar(10)
# Yes, we're getting an exception but with the wrong value: bug found!
def test6():
with assert_raises_regexp(ValueError, "arg should be less than 10"):
bar(10)
test5 which uses @raises will pass, but it will pass for the wrong reason. test6 performs a finer test which reveals that the ValueError raised was not the one we wanted.
I’m testing exceptions with nose. Here’s an example:
def testDeleteUserUserNotFound(self):
"Test exception is raised when trying to delete non-existent users"
try:
self.client.deleteUser('10000001-0000-0000-1000-100000000000')
# make nose fail here
except UserNotFoundException:
assert True
The assert is executed if the exception is raised, but if no exception is raised, it won’t be executed.
Is there anything I can put on the commented line above so that if no exception is raised nose will report a failure?
I don’t know what nose is, but have you tried using ‘else’ after the except clause. I.e.
else:
assert False
def testDeleteUserUserNotFound(self):
"Test exception is raised when trying to delete non-existent users"
try:
self.client.deleteUser('10000001-0000-0000-1000-100000000000')
assert False # <---
except UserNotFoundException:
assert True
The semantics of try/except imply that the flow of execution leaves the try block on an exception, so assert False will not run if an exception is raised. Also, execution will not re-enter the try block again after the except block is done running, so you shouldn’t run into trouble.
↓
(statements)
↓ exception
(try) ↚──────────→ (except)
↓ │
(statements) ←───────────┘
↓
nose provides tools for testing exceptions (like unittest does).
Try this example (and read about the other tools at Nose Testing Tools
from nose.tools import *
l = []
d = dict()
@raises(Exception)
def test_Exception1():
'''this test should pass'''
l.pop()
@raises(KeyError)
def test_Exception2():
'''this test should pass'''
d[1]
@raises(KeyError)
def test_Exception3():
'''this test should fail (IndexError raised but KeyError was expected)'''
l.pop()
def test_Exception4():
'''this test should fail with KeyError'''
d[1]
I would think that this is the proper way that you were looking for because it lets you be specific about the exceptions that you expect or want. So you actually provoke the error to see that it raises the right exception. And then you let nose evaluate the result. (Put as little logic into the unit tests as possible!)
I do not know why it is not here yet but exist one more way:
import unittest
class TestCase(unittest.TestCase):
def testKeyError(self):
d = dict()
with self.assertRaises(KeyError):
d[1]
Use assert_raises:
from nose.tools import assert_raises
our_method = self.client.deleteUser
arg1 = '10000001-0000-0000-1000-100000000000'
expected_exception = UserNotFoundException
assert_raises(expected_exception, our_method, arg1)
Using try and catch in your tests seems like bad practice (most cases).
There’s no specific documentation in nose because it’s basically just a wrapper around unittest.TestCase.assertRaises (ref. How to use nose's assert_raises?)
I strongly recommend using assert_raises and assert_raises_regexp from nose.tools, which duplicate the behavior of assertRaises and assertRaisesRegexp from unittest.TestCase. These allow using the same functionality as provided by unittest.TestCase in test suites that do not actually use the unittest.TestCase class.
I find that @raises is much too blunt an instrument. Here is code illustrating the problem:
from nose.tools import *
something = ["aaa", "bbb"]
def foo(x, source=None):
if source is None:
source = something
return source[x]
# This is fine
@raises(IndexError)
def test1():
foo(3)
# This is fine. The expected error does not happen because we made
# a mistake in the test or in the code. The failure indicates we made
# a mistake.
@raises(IndexError)
def test2():
foo(1)
# This passes for the wrong reasons.
@raises(IndexError)
def test3():
source = something[2] # This is the line that raises the exception.
foo(10, source) # This is not tested.
# When we use assert_raises, we can isolate the line where we expect
# the failure. This causes an error due to the exception raised in
# the first line of the function.
def test4():
source = something[2]
with assert_raises(IndexError):
foo(10, source)
test3 passes, but not because foo has raised the exception we were expecting but because the code that sets up the data to be used by foo fails with the same exception. test4 shows how the test can be written using assert_raises to actually test what we mean to be testing. The problem on the first line will cause Nose to report an error and then we can rewrite the test so that that line so that we can finally test what we did mean to test.
@raises does not allow testing the message associated with the exception. When I raise ValueError, just to take one example, I usually want to raise it with an informative message. Here’s an example:
def bar(arg):
if arg: # This is incorrect code.
raise ValueError("arg should be higher than 3")
if arg >= 10:
raise ValueError("arg should be less than 10")
# We don't know which of the possible `raise` statements was reached.
@raises(ValueError)
def test5():
bar(10)
# Yes, we're getting an exception but with the wrong value: bug found!
def test6():
with assert_raises_regexp(ValueError, "arg should be less than 10"):
bar(10)
test5 which uses @raises will pass, but it will pass for the wrong reason. test6 performs a finer test which reveals that the ValueError raised was not the one we wanted.