Test code and branch coverage simultanously with Pytest

Question

I am using pytest to test my Python code.

To test for code coverage (C0 coverage) I run pytest --cov and I can specify my desired coverage in my pyproject.toml file like this:

[tool.coverage.report]
fail_under = 95

I get this result with a coverage a 96.30%:

---------- coverage: platform linux, python 3.8.13-final-0 -----------
Name                                                                         Stmts   Miss  Cover
------------------------------------------------------------------------------------------------

.....................................

Required test coverage of 95.0% reached. Total coverage: 96.30%

To test for branch coverage (C1 coverage) I run pytest --cov --cov-branch. I get this result with a coverage of 95.44%:

---------- coverage: platform linux, python 3.8.13-final-0 -----------
Name                                                                         Stmts   Miss Branch BrPart  Cover
--------------------------------------------------------------------------------------------------------------
.....................................

Required test coverage of 95.0% reached. Total coverage: 95.44%

I get two different coverage values, so I am testing two different coverage instances. What I would like to do is be able to test for code coverage AND branch coverage with the same command, and also be able to specify two different required coverages.

For now, all I can do is execute pytest two times, with two disadvantages:

I have to run my tests 2 times, so it takes twice as long.
I am limited to the same required coverage for both.

Asked By: Kins

||

Source

Answer 1

My final solution uses pytest to execute the tests, coverage to generate coverage reports and coverage-threshold to interpret the results.

From coverage-theshold‘s README:

A command line tool for checking coverage reports against configurable coverage minimums. Currently built for use around python’s coverage.

Tools to install are: pytest (you can also use another test runner), coverage, coverage-threshold.

I execute:

coverage run --branch -m pytest src/  # the --branch argument enables branch coverage
coverage json
coverage-threshold

My pyproject.toml contains the following lines to configure:

[coverage-threshold]
line_coverage_min = 90
file_line_coverage_min = 90
branch_coverage_min = 80
file_branch_coverage_min = 80

In order:

line_coverage_min sets a threshold for the line coverage.
file_line_coverage_min sets a threshold for the line coverage to be respected by each file.
branch_coverage_min sets a threshold for the branch coverage.
file_branch_coverage_min sets a threshold for the branch coverage to be respected by each file.

The CLI options are:

> coverage-threshold --help
usage: coverage-threshold [-h] [--line-coverage-min LINE_COVERAGE_MIN]
                          [--branch-coverage-min BRANCH_COVERAGE_MIN]
                          [--combined-coverage-min COMBINED_COVERAGE_MIN]
                          [--file-line-coverage-min FILE_LINE_COVERAGE_MIN]
                          [--file-branch-coverage-min FILE_BRANCH_COVERAGE_MIN]
                          [--file-combined-coverage-min FILE_COMBINED_COVERAGE_MIN]
                          [--coverage-json COVERAGE_JSON] [--config CONFIG]

A command line tool for checking coverage reports against configurable coverage minimums

optional arguments:
  -h, --help            show this help message and exit
  --line-coverage-min LINE_COVERAGE_MIN
                        minimum global average line coverage threshold
  --branch-coverage-min BRANCH_COVERAGE_MIN
                        minimum global average branch coverage threshold
  --combined-coverage-min COMBINED_COVERAGE_MIN
                        minimum global average combined line and branch coverage threshold
  --file-line-coverage-min FILE_LINE_COVERAGE_MIN
                        the line coverage threshold for each file
  --file-branch-coverage-min FILE_BRANCH_COVERAGE_MIN
                        the branch coverage threshold for each file
  --file-combined-coverage-min FILE_COMBINED_COVERAGE_MIN
                        the combined line and branch coverage threshold for each file
  --coverage-json COVERAGE_JSON
                        path to coverage json (default: ./coverage.json)
  --config CONFIG       path to config file (default: ./pyproject.toml)

Answered By: Kins

Test code and branch coverage simultanously with Pytest

Question:

Answers: