How to properly assert that an exception gets raised in pytest?
Question:
Code:
# coding=utf-8
import pytest
def whatever():
return 9/0
def test_whatever():
try:
whatever()
except ZeroDivisionError as exc:
pytest.fail(exc, pytrace=True)
Output:
================================ test session starts =================================
platform linux2 -- Python 2.7.3 -- py-1.4.20 -- pytest-2.5.2
plugins: django, cov
collected 1 items
pytest_test.py F
====================================== FAILURES ======================================
___________________________________ test_whatever ____________________________________
def test_whatever():
try:
whatever()
except ZeroDivisionError as exc:
> pytest.fail(exc, pytrace=True)
E Failed: integer division or modulo by zero
pytest_test.py:12: Failed
============================== 1 failed in 1.16 seconds ==============================
How to make pytest print traceback, so I would see where in the whatever
function an exception was raised?
Answers:
Do you mean something like this:
def test_raises():
with pytest.raises(Exception) as exc_info:
raise Exception('some info')
# these asserts are identical; you can use either one
assert exc_info.value.args[0] == 'some info'
assert str(exc_info.value) == 'some info'
pytest.raises(Exception)
is what you need.
Code
import pytest
def test_passes():
with pytest.raises(Exception) as e_info:
x = 1 / 0
def test_passes_without_info():
with pytest.raises(Exception):
x = 1 / 0
def test_fails():
with pytest.raises(Exception) as e_info:
x = 1 / 1
def test_fails_without_info():
with pytest.raises(Exception):
x = 1 / 1
# Don't do this. Assertions are caught as exceptions.
def test_passes_but_should_not():
try:
x = 1 / 1
assert False
except Exception:
assert True
# Even if the appropriate exception is caught, it is bad style,
# because the test result is less informative
# than it would be with pytest.raises(e)
# (it just says pass or fail.)
def test_passes_but_bad_style():
try:
x = 1 / 0
assert False
except ZeroDivisionError:
assert True
def test_fails_but_bad_style():
try:
x = 1 / 1
assert False
except ZeroDivisionError:
assert True
Output
============================================================================================= test session starts ==============================================================================================
platform linux2 -- Python 2.7.6 -- py-1.4.26 -- pytest-2.6.4
collected 7 items
test.py ..FF..F
=================================================================================================== FAILURES ===================================================================================================
__________________________________________________________________________________________________ test_fails __________________________________________________________________________________________________
def test_fails():
with pytest.raises(Exception) as e_info:
> x = 1 / 1
E Failed: DID NOT RAISE
test.py:13: Failed
___________________________________________________________________________________________ test_fails_without_info ____________________________________________________________________________________________
def test_fails_without_info():
with pytest.raises(Exception):
> x = 1 / 1
E Failed: DID NOT RAISE
test.py:17: Failed
___________________________________________________________________________________________ test_fails_but_bad_style ___________________________________________________________________________________________
def test_fails_but_bad_style():
try:
x = 1 / 1
> assert False
E assert False
test.py:43: AssertionError
====================================================================================== 3 failed, 4 passed in 0.02 seconds ======================================================================================
Note that e_info
saves the exception object so you can extract details from it. For example, if you want to check the exception call stack or another nested exception inside.
Have you tried to remove “pytrace=True” ?
pytest.fail(exc, pytrace=True) # before
pytest.fail(exc) # after
Have you tried to run with ‘–fulltrace’ ?
Better practice will be using a class that inherit unittest.TestCase and running self.assertRaises.
For example:
import unittest
def whatever():
return 9/0
class TestWhatEver(unittest.TestCase):
def test_whatever():
with self.assertRaises(ZeroDivisionError):
whatever()
Then you would execute it by running:
pytest -vs test_path
you can try
def test_exception():
with pytest.raises(Exception) as excinfo:
function_that_raises_exception()
assert str(excinfo.value) == 'some info'
There are two ways to handle these kind of cases in pytest:
-
Using pytest.raises
function
-
Using pytest.mark.xfail
decorator
As the documentation says:
Using pytest.raises
is likely to be better for cases where you are testing exceptions your own code is deliberately raising, whereas using @pytest.mark.xfail
with a check function is probably better for something like documenting unfixed bugs (where the test describes what “should” happen) or bugs in dependencies.
Usage of pytest.raises
:
def whatever():
return 9/0
def test_whatever():
with pytest.raises(ZeroDivisionError):
whatever()
Usage of pytest.mark.xfail
:
@pytest.mark.xfail(raises=ZeroDivisionError)
def test_whatever():
whatever()
Output of pytest.raises
:
============================= test session starts ============================
platform linux2 -- Python 2.7.10, pytest-3.2.3, py-1.4.34, pluggy-0.4.0 --
/usr/local/python_2.7_10/bin/python
cachedir: .cache
rootdir: /home/user, inifile:
collected 1 item
test_fun.py::test_whatever PASSED
======================== 1 passed in 0.01 seconds =============================
Output of pytest.xfail
marker:
============================= test session starts ============================
platform linux2 -- Python 2.7.10, pytest-3.2.3, py-1.4.34, pluggy-0.4.0 --
/usr/local/python_2.7_10/bin/python
cachedir: .cache
rootdir: /home/user, inifile:
collected 1 item
test_fun.py::test_whatever xfail
======================== 1 xfailed in 0.03 seconds=============================
This solution is what we are using:
def test_date_invalidformat():
"""
Test if input incorrect data will raises ValueError exception
"""
date = "06/21/2018 00:00:00"
with pytest.raises(ValueError):
app.func(date) #my function to be tested
Please refer to pytest, https://docs.pytest.org/en/latest/reference.html#pytest-raises
Right way is using pytest.raises
but I found interesting alternative way in comments here and want to save it for future readers of this question:
try:
thing_that_rasises_typeerror()
assert False
except TypeError:
assert True
pytest constantly evolves and with one of the nice changes in the recent past it is now possible to simultaneously test for
- the exception type (strict test)
- the error message (strict or loose check using a regular expression)
Two examples from the documentation:
with pytest.raises(ValueError, match='must be 0 or None'):
raise ValueError('value must be 0 or None')
with pytest.raises(ValueError, match=r'must be d+$'):
raise ValueError('value must be 42')
I have been using that approach in a number of projects and like it very much.
Note:
This comment by ilya-rusin also suggests aforementioned approach.
There are two ways to handle exceptions in pytest
:
- Using
pytest.raises
to write assertions about raised exceptions
- Using
@pytest.mark.xfail
1. Using pytest.raises
From the docs:
In order to write assertions about raised exceptions, you can use pytest.raises
as a context manager
Examples:
Asserting just an exception:
import pytest
def test_zero_division():
with pytest.raises(ZeroDivisionError):
1 / 0
with pytest.raises(ZeroDivisionError)
says that whatever is
in the next block of code should raise a ZeroDivisionError
exception. If no exception is raised, the test fails. If the test raises a different exception, it fails.
If you need to have access to the actual exception info:
import pytest
def f():
f()
def test_recursion_depth():
with pytest.raises(RuntimeError) as excinfo:
f()
assert "maximum recursion" in str(excinfo.value)
excinfo
is a ExceptionInfo
instance, which is a wrapper around the actual exception raised. The main attributes of interest are .type
, .value
and .traceback
.
2. Using @pytest.mark.xfail
It is also possible to specify a raises
argument to pytest.mark.xfail
.
import pytest
@pytest.mark.xfail(raises=IndexError)
def test_f():
l = [1, 2, 3]
l[10]
@pytest.mark.xfail(raises=IndexError)
says that whatever is
in the next block of code should raise an IndexError
exception. If an IndexError
is raised, test is marked as xfailed (x)
. If no exception is raised, the test is marked as xpassed (X)
. If the test raises a different exception, it fails.
Notes:
-
Using pytest.raises
is likely to be better for cases where you are testing exceptions your own code is deliberately raising, whereas using @pytest.mark.xfail
with a check function is probably better for something like documenting unfixed bugs or bugs in dependencies.
-
You can pass a match
keyword parameter to the context-manager (pytest.raises
) to test that a regular expression matches on the string representation of an exception. (see more)
If you want to test for a specific error type, use a combination of try, catch and raise:
#-- test for TypeError
try:
myList.append_number("a")
assert False
except TypeError: pass
except: assert False
The top answers submitted here are useful if you are expecting an exception to be raised for your test-case. It’s not very useful if your test may raise an exception and you’d want to handle it with grace in either scenario.
If you have a test-case that may (not will) raise exception, I think this might be a better option.
@python.mark.parametrize("request_query, response_code", query_response_dataset)
def test_big_query_submission(request_query, response_code):
try:
stats = bigquery.Client().query(request_query)
except Exception as e:
assert False, f"Exception raised: {e}"
assert stats.errors is None
This way you are covered to gracefully fail a test instead of crashing a test through a raised exception for whatever reason.
i just wrote a hook every test gets
the hook :
@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item: Item, call: CallInfo):
outcome = yield # The result after the test is completed
result = outcome.get_result()
if result.when == "call":
if result.failed == True:
else:
i have a function that updates a document with test result and adds the trace by using : result.longrepr.reprcrash.message
i Don’t know if this is the best way , but it does answer the question of how to print the traceback using only pytest
@Obviously i have some other code
Just adding another "dumb" suggestion as I don’t see it in the existing answers. Essentially you can initialise an error variable as None
, do-the-thing in a try/except block, and then check the class/value of the error variable after that
e = None
try:
blah()
except Exception as exc:
e = exc
assert e.__class__ == ValueError # or whatever you expect
assert str(e) == "expected message"
Code:
# coding=utf-8
import pytest
def whatever():
return 9/0
def test_whatever():
try:
whatever()
except ZeroDivisionError as exc:
pytest.fail(exc, pytrace=True)
Output:
================================ test session starts =================================
platform linux2 -- Python 2.7.3 -- py-1.4.20 -- pytest-2.5.2
plugins: django, cov
collected 1 items
pytest_test.py F
====================================== FAILURES ======================================
___________________________________ test_whatever ____________________________________
def test_whatever():
try:
whatever()
except ZeroDivisionError as exc:
> pytest.fail(exc, pytrace=True)
E Failed: integer division or modulo by zero
pytest_test.py:12: Failed
============================== 1 failed in 1.16 seconds ==============================
How to make pytest print traceback, so I would see where in the whatever
function an exception was raised?
Do you mean something like this:
def test_raises():
with pytest.raises(Exception) as exc_info:
raise Exception('some info')
# these asserts are identical; you can use either one
assert exc_info.value.args[0] == 'some info'
assert str(exc_info.value) == 'some info'
pytest.raises(Exception)
is what you need.
Code
import pytest
def test_passes():
with pytest.raises(Exception) as e_info:
x = 1 / 0
def test_passes_without_info():
with pytest.raises(Exception):
x = 1 / 0
def test_fails():
with pytest.raises(Exception) as e_info:
x = 1 / 1
def test_fails_without_info():
with pytest.raises(Exception):
x = 1 / 1
# Don't do this. Assertions are caught as exceptions.
def test_passes_but_should_not():
try:
x = 1 / 1
assert False
except Exception:
assert True
# Even if the appropriate exception is caught, it is bad style,
# because the test result is less informative
# than it would be with pytest.raises(e)
# (it just says pass or fail.)
def test_passes_but_bad_style():
try:
x = 1 / 0
assert False
except ZeroDivisionError:
assert True
def test_fails_but_bad_style():
try:
x = 1 / 1
assert False
except ZeroDivisionError:
assert True
Output
============================================================================================= test session starts ==============================================================================================
platform linux2 -- Python 2.7.6 -- py-1.4.26 -- pytest-2.6.4
collected 7 items
test.py ..FF..F
=================================================================================================== FAILURES ===================================================================================================
__________________________________________________________________________________________________ test_fails __________________________________________________________________________________________________
def test_fails():
with pytest.raises(Exception) as e_info:
> x = 1 / 1
E Failed: DID NOT RAISE
test.py:13: Failed
___________________________________________________________________________________________ test_fails_without_info ____________________________________________________________________________________________
def test_fails_without_info():
with pytest.raises(Exception):
> x = 1 / 1
E Failed: DID NOT RAISE
test.py:17: Failed
___________________________________________________________________________________________ test_fails_but_bad_style ___________________________________________________________________________________________
def test_fails_but_bad_style():
try:
x = 1 / 1
> assert False
E assert False
test.py:43: AssertionError
====================================================================================== 3 failed, 4 passed in 0.02 seconds ======================================================================================
Note that e_info
saves the exception object so you can extract details from it. For example, if you want to check the exception call stack or another nested exception inside.
Have you tried to remove “pytrace=True” ?
pytest.fail(exc, pytrace=True) # before
pytest.fail(exc) # after
Have you tried to run with ‘–fulltrace’ ?
Better practice will be using a class that inherit unittest.TestCase and running self.assertRaises.
For example:
import unittest
def whatever():
return 9/0
class TestWhatEver(unittest.TestCase):
def test_whatever():
with self.assertRaises(ZeroDivisionError):
whatever()
Then you would execute it by running:
pytest -vs test_path
you can try
def test_exception():
with pytest.raises(Exception) as excinfo:
function_that_raises_exception()
assert str(excinfo.value) == 'some info'
There are two ways to handle these kind of cases in pytest:
-
Using
pytest.raises
function -
Using
pytest.mark.xfail
decorator
As the documentation says:
Using
pytest.raises
is likely to be better for cases where you are testing exceptions your own code is deliberately raising, whereas using@pytest.mark.xfail
with a check function is probably better for something like documenting unfixed bugs (where the test describes what “should” happen) or bugs in dependencies.
Usage of pytest.raises
:
def whatever():
return 9/0
def test_whatever():
with pytest.raises(ZeroDivisionError):
whatever()
Usage of pytest.mark.xfail
:
@pytest.mark.xfail(raises=ZeroDivisionError)
def test_whatever():
whatever()
Output of pytest.raises
:
============================= test session starts ============================
platform linux2 -- Python 2.7.10, pytest-3.2.3, py-1.4.34, pluggy-0.4.0 --
/usr/local/python_2.7_10/bin/python
cachedir: .cache
rootdir: /home/user, inifile:
collected 1 item
test_fun.py::test_whatever PASSED
======================== 1 passed in 0.01 seconds =============================
Output of pytest.xfail
marker:
============================= test session starts ============================
platform linux2 -- Python 2.7.10, pytest-3.2.3, py-1.4.34, pluggy-0.4.0 --
/usr/local/python_2.7_10/bin/python
cachedir: .cache
rootdir: /home/user, inifile:
collected 1 item
test_fun.py::test_whatever xfail
======================== 1 xfailed in 0.03 seconds=============================
This solution is what we are using:
def test_date_invalidformat():
"""
Test if input incorrect data will raises ValueError exception
"""
date = "06/21/2018 00:00:00"
with pytest.raises(ValueError):
app.func(date) #my function to be tested
Please refer to pytest, https://docs.pytest.org/en/latest/reference.html#pytest-raises
Right way is using pytest.raises
but I found interesting alternative way in comments here and want to save it for future readers of this question:
try:
thing_that_rasises_typeerror()
assert False
except TypeError:
assert True
pytest constantly evolves and with one of the nice changes in the recent past it is now possible to simultaneously test for
- the exception type (strict test)
- the error message (strict or loose check using a regular expression)
Two examples from the documentation:
with pytest.raises(ValueError, match='must be 0 or None'):
raise ValueError('value must be 0 or None')
with pytest.raises(ValueError, match=r'must be d+$'):
raise ValueError('value must be 42')
I have been using that approach in a number of projects and like it very much.
Note:
This comment by ilya-rusin also suggests aforementioned approach.
There are two ways to handle exceptions in pytest
:
- Using
pytest.raises
to write assertions about raised exceptions - Using
@pytest.mark.xfail
1. Using pytest.raises
From the docs:
In order to write assertions about raised exceptions, you can use
pytest.raises
as a context manager
Examples:
Asserting just an exception:
import pytest
def test_zero_division():
with pytest.raises(ZeroDivisionError):
1 / 0
with pytest.raises(ZeroDivisionError)
says that whatever is
in the next block of code should raise a ZeroDivisionError
exception. If no exception is raised, the test fails. If the test raises a different exception, it fails.
If you need to have access to the actual exception info:
import pytest
def f():
f()
def test_recursion_depth():
with pytest.raises(RuntimeError) as excinfo:
f()
assert "maximum recursion" in str(excinfo.value)
excinfo
is a ExceptionInfo
instance, which is a wrapper around the actual exception raised. The main attributes of interest are .type
, .value
and .traceback
.
2. Using @pytest.mark.xfail
It is also possible to specify a raises
argument to pytest.mark.xfail
.
import pytest
@pytest.mark.xfail(raises=IndexError)
def test_f():
l = [1, 2, 3]
l[10]
@pytest.mark.xfail(raises=IndexError)
says that whatever is
in the next block of code should raise an IndexError
exception. If an IndexError
is raised, test is marked as xfailed (x)
. If no exception is raised, the test is marked as xpassed (X)
. If the test raises a different exception, it fails.
Notes:
Using
pytest.raises
is likely to be better for cases where you are testing exceptions your own code is deliberately raising, whereas using@pytest.mark.xfail
with a check function is probably better for something like documenting unfixed bugs or bugs in dependencies.You can pass a
match
keyword parameter to the context-manager (pytest.raises
) to test that a regular expression matches on the string representation of an exception. (see more)
If you want to test for a specific error type, use a combination of try, catch and raise:
#-- test for TypeError
try:
myList.append_number("a")
assert False
except TypeError: pass
except: assert False
The top answers submitted here are useful if you are expecting an exception to be raised for your test-case. It’s not very useful if your test may raise an exception and you’d want to handle it with grace in either scenario.
If you have a test-case that may (not will) raise exception, I think this might be a better option.
@python.mark.parametrize("request_query, response_code", query_response_dataset)
def test_big_query_submission(request_query, response_code):
try:
stats = bigquery.Client().query(request_query)
except Exception as e:
assert False, f"Exception raised: {e}"
assert stats.errors is None
This way you are covered to gracefully fail a test instead of crashing a test through a raised exception for whatever reason.
i just wrote a hook every test gets
the hook :
@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item: Item, call: CallInfo):
outcome = yield # The result after the test is completed
result = outcome.get_result()
if result.when == "call":
if result.failed == True:
else:
i have a function that updates a document with test result and adds the trace by using : result.longrepr.reprcrash.message
i Don’t know if this is the best way , but it does answer the question of how to print the traceback using only pytest
@Obviously i have some other code
Just adding another "dumb" suggestion as I don’t see it in the existing answers. Essentially you can initialise an error variable as None
, do-the-thing in a try/except block, and then check the class/value of the error variable after that
e = None
try:
blah()
except Exception as exc:
e = exc
assert e.__class__ == ValueError # or whatever you expect
assert str(e) == "expected message"