pytest coverage not working after python update to 3.11

Question:

Issue:
pytest stopped generating coverage

Description:
I upgraded python version from 3.6 to 3.11 (Ubuntu 18.04). I followed this https://www.debugpoint.com/install-python-3-11-ubuntu/ in order to do that.
Had issues with pip, followed this advice: https://stackoverflow.com/a/72685573/15923186

I have a project with following structure:

test.sh
|-sorting
  |- env (the virtual environment for sorting)
  |- requirements.txt
  |- pyproject.toml
  |- __init__.py
  |- merge_sort.py
  |- tests
     |-__init__.py
     |-test_merge_sort.py
  
|-something_else (etc.)

requirements.txt are following:

attrs==22.1.0
black==22.8.0
click==8.0.4
coverage==6.2
flake8==5.0.4
importlib-metadata==4.2.0
iniconfig==1.1.1
mccabe==0.7.0
mypy-extensions==0.4.3
packaging==21.3
pathspec==0.9.0
platformdirs==2.4.0
pluggy==1.0.0
py==1.11.0
pycodestyle==2.9.1
pyflakes==2.5.0
pyparsing==3.0.9
pytest==7.0.1
pytest-cov==4.0.0
pytest-cover==3.0.0
tomli==1.2.3
typing-extensions==4.1.1
zipp==3.6.0

Pyproject is following:

[tool.pytest.ini_options]
pythonpath = [
  "."
]

The test script is following:

#!/bin/bash

source "$1"/env/bin/activate

cd "$1" || { echo "Project not found!"; exit 1;}

pytest -vv --cov="$1" --cov-report xml --cov-report term

And is invoked like this:
./test.sh sorting

I’m not "reusing" old virtual environments, I double checked. Before running tests I prepared a new one like this (the same as previously when using python 3.6):

python3 -m venv env
source env/bin/activate
pip install -r requirements.txt

Python 3.11 is the default one for python3 command (output below):

> Python 3.11.0 (main, Oct 24 2022, 19:56:01) [GCC 7.5.0] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> 

Result:
Nothing displayed, no coverage report generated

Expected result:
Coverage report generated and displayed in the terminal

I also tried to run with:

python3 -m pytest -vv --cov="$1" --cov-report xml --cov-report term

No difference.

If I remove

 --cov="$1" --cov-report xml --cov-report term

from the test.sh script I get following output (which is good and expected)

test session starts ================================================================
platform linux -- Python 3.11.0, pytest-7.0.1, pluggy-1.0.0 -- /path/to/the/env/which/is/irrelevant/sorting/env/bin/python3
cachedir: .pytest_cache
rootdir: /home/mysusrname/folder_name/subfolder_name/sorting, configfile: pyproject.toml
plugins: cov-4.0.0
collected 3 items                                                                                                                                  

tests/test_merge_sort.py::test_merge_sort[numbers0] PASSED                                                                                   [ 33%]
tests/test_merge_sort.py::test_merge_sort[numbers1] PASSED                                                                                   [ 66%]
tests/test_merge_sort.py::test_merge_sort[numbers2] PASSED                                                                                   [100%]

3 passed in 0.01s =================================================================
Asked By: Gameplay

||

Answers:

According to your requirement.txt, you are using pytest 7.0.1 which came out before python 3.11 (https://docs.pytest.org/en/stable/changelog.html#pytest-7-0-1-2022-02-11), which makes it unlikely to support it as it didn’t exist at that time. (N-B: sometimes it works, but it is not officially supported by the developers of pytest)

For your case, as you need to upgrade several Python version at the time, I would use the following process to upgrade Python version and dependencies.

  • In Python 3.6, upgrade all dependencies to the latest version that supports Python 3.6
  • Test
  • Upgrade to Python 3.7
  • Test
  • In Python 3.7, upgrade all dependencies to the latest version that supports Python 3.7
  • Test
  • Keep going

It is of course possible to skip some python version to speed up the process, but you might run into the same issues as you encountered.

Usually, there are between 3 and 4 supported version of Python out at the same time (https://endoflife.date/python), which means that most library will support those versions (not always the case, for example numpy support less versions: https://numpy.org/neps/nep-0029-deprecation_policy.html), so it is usually possible to skip 1 version or 2. 3 is likely to not work.

Answered By: Nathan

So I couldn’t agree with @Nathan and kept digging and actually my first comment is right and seems to be faster and imho cleaner solution.

After upgrading python like I described and having a requirements.txt with fixed versions:

  • delete the old virtual environment

  • remove versions from requirements.txt (example below):

      attrs
      black
      click
      coverage
      flake8
      importlib-metadata
      iniconfig
      mccabe
      mypy-extensions
      packaging
      pathspec
      platformdirs
      pluggy
      py
      pycodestyle
      pyflakes
      pyparsing
      pytest
      pytest-cov
      pytest-cover
      tomli
      typing_extensions
      zipp
    
  • after creating a "fresh" venv and sourcing it run pip install --upgrade pip to make sure the pip is up to date

  • install the packages (utilizing the requirements.txt with NO versions specified in the venv mentioned above) pip install -r requirements.txt

  • the newest versions supported for your python distro will be used in such case

  • run pip freeze > requirements.txt to overwrite the requirements file and again have stable version of it with versions of libraries fixed (example corresponding "clean" file below)

      attrs==22.1.0
      black==22.12.0
      click==8.1.3
      coverage==6.5.0
      flake8==6.0.0
      importlib-metadata==5.1.0
      iniconfig==1.1.1
      mccabe==0.7.0
      mypy-extensions==0.4.3
      packaging==22.0
      pathspec==0.10.3
      platformdirs==2.6.0
      pluggy==1.0.0
      py==1.11.0
      pycodestyle==2.10.0
      pyflakes==3.0.1
      pyparsing==3.0.9
      pytest==7.2.0
      pytest-cov==4.0.0
      pytest-cover==3.0.0
      tomli==2.0.1
      typing_extensions==4.4.0
      zipp==3.11.0
    

Side notes:
I don’t think having multiple versions of python is convenient to use, nor it is required in most cases actually. Probably one version as a "default"/to go and maybe a new release / release candidate as a second one is enough. Cannot imagine a situation where I need 5 or 6 versions…The venvs manage versions of python libraries and e.g. docker containers (or other virtualization method) can abstract anything else (not necessarily python related).
Not to be misunderstood:
Is @Nathan ‘s solution bad? Nope, it’s good, but there are some points I disagree with, that’s all. Also his comments helped me out to come up with a solution of my own, so upvote from me 🙂

Answered By: Gameplay