How to run different Pytest arguments or marks from VS Code test runner interface?
Question:
I’m having trouble getting the VS Code PyTest code runner to work the way I’d like. It seems pytest options may be an all-or-nothing situation.
Is there any way to run different sets of PyTest options easily in the VS Code interface?
For example:
-
By default, run all tests not marked with @pytest.mark.slow.
- This can be done with the argument
-m "not slow"
- But, if I put that in a pytest.ini file, then it will never run any tests marked slow, even if I pick that particular test in the interface and try to run it. The resulting output is
collected 1 item... 1 item deselected.
-
Run sometimes with coverage enabled, and sometimes without.
The only way I can see to do this is to run PyTest from the command line, which then loses the benefit of auto-discovery, running/debugging individual tests from the in-line interface, etc.
What am I missing?
Note: Currently using VS Code 1.45.1, Python 3.7.6, and PyTest 5.3.5
Answers:
You’re not missing anything. There currently isn’t a way to provide per-execution arguments to get the integration you want with the Test Explorer.
You can’t do this directly in the Test interface (still, as of Jan 2023), but you can do it with commandline arguments or separate scripts that you run at a terminal.
So it forces you to remember some parts of pytest’s commandline interface, or write script(s) to run pytest exactly how you want for different use cases, but that’s the best I have figured out so far without updating vs code or an extension. I don’t know a way to update the Test Explorer interface with the results of those runs, however.
For example, you can put the following into your pytest.ini file, which will setup all the coverage flags, but run by default without coverage enabled, but just adding the --cov flag will run all the tests with coverage:
pytest.ini:
[pytest]
minversion = 6.0
python_files = test*.py
python_classes = Test
python_functions = test_* *_test
testpaths = ./
addopts = --cov-branch --cov-report html --cov-report xml:coverage.xml --cov-report term --ignore=<folder to ignore when discovering tests>
With the above, if one runs:
pytest
at the terminal from the repo root, pytest will run all tests without coverage, but using:
pytest --cov
it will include coverage as well, applying all the coverage flags in the addopts property in the ini file. This assumes you have installed and configured all the appropriate plugins and files for coverage.
Similarly, one could use the other pytest commandline args to run only certain test file marks, folders, etc, or create bash scripts that allow consolidating different types of tests or sub-suites into specific commands.
I’m having trouble getting the VS Code PyTest code runner to work the way I’d like. It seems pytest options may be an all-or-nothing situation.
Is there any way to run different sets of PyTest options easily in the VS Code interface?
For example:
-
By default, run all tests not marked with
@pytest.mark.slow.- This can be done with the argument
-m "not slow" - But, if I put that in a pytest.ini file, then it will never run any tests marked slow, even if I pick that particular test in the interface and try to run it. The resulting output is
collected 1 item... 1 item deselected.
- This can be done with the argument
-
Run sometimes with coverage enabled, and sometimes without.
The only way I can see to do this is to run PyTest from the command line, which then loses the benefit of auto-discovery, running/debugging individual tests from the in-line interface, etc.
What am I missing?
Note: Currently using VS Code 1.45.1, Python 3.7.6, and PyTest 5.3.5
You’re not missing anything. There currently isn’t a way to provide per-execution arguments to get the integration you want with the Test Explorer.
You can’t do this directly in the Test interface (still, as of Jan 2023), but you can do it with commandline arguments or separate scripts that you run at a terminal.
So it forces you to remember some parts of pytest’s commandline interface, or write script(s) to run pytest exactly how you want for different use cases, but that’s the best I have figured out so far without updating vs code or an extension. I don’t know a way to update the Test Explorer interface with the results of those runs, however.
For example, you can put the following into your pytest.ini file, which will setup all the coverage flags, but run by default without coverage enabled, but just adding the --cov flag will run all the tests with coverage:
pytest.ini:
[pytest]
minversion = 6.0
python_files = test*.py
python_classes = Test
python_functions = test_* *_test
testpaths = ./
addopts = --cov-branch --cov-report html --cov-report xml:coverage.xml --cov-report term --ignore=<folder to ignore when discovering tests>
With the above, if one runs:
pytest
at the terminal from the repo root, pytest will run all tests without coverage, but using:
pytest --cov
it will include coverage as well, applying all the coverage flags in the addopts property in the ini file. This assumes you have installed and configured all the appropriate plugins and files for coverage.
Similarly, one could use the other pytest commandline args to run only certain test file marks, folders, etc, or create bash scripts that allow consolidating different types of tests or sub-suites into specific commands.