Python – doctest vs. unittest
Question:
I’m trying to get started with unit testing in Python and I was wondering if someone could explain the advantages and disadvantages of doctest and unittest.
What conditions would you use each for?
Answers:
I use unittest almost exclusively.
Once in a while, I’ll put some stuff in a docstring that’s usable by doctest.
95% of the test cases are unittest.
Why? I like keeping docstrings somewhat shorter and more to the point. Sometimes test cases help clarify a docstring. Most of the time, the application’s test cases are too long for a docstring.
If you’re just getting started with the idea of unit testing, I would start with doctest because it is so simple to use. It also naturally provides some level of documentation. And for more comprehensive testing with doctest, you can place tests in an external file so it doesn’t clutter up your documentation.
I would suggest unittest if you’re coming from a background of having used JUnit or something similar, where you want to be able to write unit tests in generally the same way as you have been elsewhere.
I almost never use doctests. I want my code to be self documenting, and the docstrings provide the documentation to the user. IMO adding hundreds of lines of tests to a module makes the docstrings far less readable. I also find unit tests easier to modify when needed.
I use unittest exclusively; I think doctest clutters up the main module too much. This probably has to do with writing thorough tests.
Using both is a valid and rather simple option. The doctest module provides the DoctTestSuite and DocFileSuite methods which create a unittest-compatible testsuite from a module or file, respectively.
So I use both and typically use doctest for simple tests with functions that require little or no setup (simple types for arguments). I actually think a few doctest tests help document the function, rather than detract from it.
But for more complicated cases, and for a more comprehensive set of test cases, I use unittest which provides more control and flexibility.
I prefer the discovery based systems (“nose” and “py.test”, using the former currently).
doctest is nice when the test is also good as a documentation, otherwise they tend to clutter the code too much.
Both are valuable. I use both doctest and nose taking the place of unittest. I use doctest for cases where the test is giving an example of usage that is actually useful as documentation. Generally I don’t make these tests comprehensive, aiming solely for informative. I’m effectively using doctest in reverse: not to test my code is correct based on my doctest, but to check that my documentation is correct based on the code.
The reason is that I find comprehensive doctests will clutter your documentation far too much, so you will either end up with either unusable docstrings, or incomplete testing.
For actually testing the code, the goal is to thoroughly test every case, rather than illustrate what is does by example, which is a different goal which I think is better met by other frameworks.
Another advantage of doctesting is that you get to make sure your code does what your documentation says it does. After a while, software changes can make your documentation and code do different things. ๐
I work as a bioinformatician, and most of the code I write is “one time, one task” scripts, code that will be run only once or twice and that execute a single specific task.
In this situation, writing big unittests may be overkill, and doctests are an useful compromise. They are quicker to write, and since they are usually incorporated in the code, they allow to always keep an eye on how the code should behave, without having to have another file open. That’s useful when writing small script.
Also, doctests are useful when you have to pass your script to a researcher that is not expert in programming. Some people find it very difficult to understand how unittests are structured; on the other hand, doctests are simple examples of usage, so people can just copy and paste them to see how to use them.
So, to resume my answer: doctests are useful when you have to write small scripts, and when you have to pass them or show them to researchers that are not computer scientists.
I don’t use doctest as a replacement for unittest. Although they overlap a bit, the two modules don’t have the same function:
-
I use unittest as a unit testing framework, meaning it helps me determine quickly the impact of any modification on the rest of the code.
-
I use doctest as a guarantee that comments (namely docstrings) are still relevant to current version of the code.
The widely documented benefits of test driven development I get from unittest. doctest solves the far more subtle danger of having outdated comments misleading the maintenance of the code.
Doctest can some times lead to wrong result. Especially when output contains escape sequences. For example
def convert():
"""
>>> convert()
'xe0xa4x95'
"""
a = 'xe0xa4x95'
return a
import doctest
doctest.testmod()
gives
**********************************************************************
File "hindi.py", line 3, in __main__.convert
Failed example:
convert()
Expected:
'เค'
Got:
'xe0xa4x95'
**********************************************************************
1 items had failures:
1 of 1 in __main__.convert
***Test Failed*** 1 failures.
Also doesn’t check the type of the output. It just compares the output strings. For example it have made some type rational which prints just like integer if it is a whole number. Then suppose you have function which return rational. So, a doctest won’t differentiate if the output is rational whole number or a integer number.
I’m trying to get started with unit testing in Python and I was wondering if someone could explain the advantages and disadvantages of doctest and unittest.
What conditions would you use each for?
I use unittest almost exclusively.
Once in a while, I’ll put some stuff in a docstring that’s usable by doctest.
95% of the test cases are unittest.
Why? I like keeping docstrings somewhat shorter and more to the point. Sometimes test cases help clarify a docstring. Most of the time, the application’s test cases are too long for a docstring.
If you’re just getting started with the idea of unit testing, I would start with doctest because it is so simple to use. It also naturally provides some level of documentation. And for more comprehensive testing with doctest, you can place tests in an external file so it doesn’t clutter up your documentation.
I would suggest unittest if you’re coming from a background of having used JUnit or something similar, where you want to be able to write unit tests in generally the same way as you have been elsewhere.
I almost never use doctests. I want my code to be self documenting, and the docstrings provide the documentation to the user. IMO adding hundreds of lines of tests to a module makes the docstrings far less readable. I also find unit tests easier to modify when needed.
I use unittest exclusively; I think doctest clutters up the main module too much. This probably has to do with writing thorough tests.
Using both is a valid and rather simple option. The doctest module provides the DoctTestSuite and DocFileSuite methods which create a unittest-compatible testsuite from a module or file, respectively.
So I use both and typically use doctest for simple tests with functions that require little or no setup (simple types for arguments). I actually think a few doctest tests help document the function, rather than detract from it.
But for more complicated cases, and for a more comprehensive set of test cases, I use unittest which provides more control and flexibility.
I prefer the discovery based systems (“nose” and “py.test”, using the former currently).
doctest is nice when the test is also good as a documentation, otherwise they tend to clutter the code too much.
Both are valuable. I use both doctest and nose taking the place of unittest. I use doctest for cases where the test is giving an example of usage that is actually useful as documentation. Generally I don’t make these tests comprehensive, aiming solely for informative. I’m effectively using doctest in reverse: not to test my code is correct based on my doctest, but to check that my documentation is correct based on the code.
The reason is that I find comprehensive doctests will clutter your documentation far too much, so you will either end up with either unusable docstrings, or incomplete testing.
For actually testing the code, the goal is to thoroughly test every case, rather than illustrate what is does by example, which is a different goal which I think is better met by other frameworks.
Another advantage of doctesting is that you get to make sure your code does what your documentation says it does. After a while, software changes can make your documentation and code do different things. ๐
I work as a bioinformatician, and most of the code I write is “one time, one task” scripts, code that will be run only once or twice and that execute a single specific task.
In this situation, writing big unittests may be overkill, and doctests are an useful compromise. They are quicker to write, and since they are usually incorporated in the code, they allow to always keep an eye on how the code should behave, without having to have another file open. That’s useful when writing small script.
Also, doctests are useful when you have to pass your script to a researcher that is not expert in programming. Some people find it very difficult to understand how unittests are structured; on the other hand, doctests are simple examples of usage, so people can just copy and paste them to see how to use them.
So, to resume my answer: doctests are useful when you have to write small scripts, and when you have to pass them or show them to researchers that are not computer scientists.
I don’t use doctest as a replacement for unittest. Although they overlap a bit, the two modules don’t have the same function:
-
I use
unittestas a unit testing framework, meaning it helps me determine quickly the impact of any modification on the rest of the code. -
I use
doctestas a guarantee that comments (namely docstrings) are still relevant to current version of the code.
The widely documented benefits of test driven development I get from unittest. doctest solves the far more subtle danger of having outdated comments misleading the maintenance of the code.
Doctest can some times lead to wrong result. Especially when output contains escape sequences. For example
def convert():
"""
>>> convert()
'xe0xa4x95'
"""
a = 'xe0xa4x95'
return a
import doctest
doctest.testmod()
gives
**********************************************************************
File "hindi.py", line 3, in __main__.convert
Failed example:
convert()
Expected:
'เค'
Got:
'xe0xa4x95'
**********************************************************************
1 items had failures:
1 of 1 in __main__.convert
***Test Failed*** 1 failures.
Also doesn’t check the type of the output. It just compares the output strings. For example it have made some type rational which prints just like integer if it is a whole number. Then suppose you have function which return rational. So, a doctest won’t differentiate if the output is rational whole number or a integer number.