Identifying which test case is running on pytest-xdist processes
Question:
I’m executing python unit tests in parallel with pytest-forked (pytest -r sxX -n auto --cache-clear --max-worker-restart=4 --forked) and there is one test case which takes quite some time and which is running at the end while the other test case runners/CPU cores are idle (because presumably there’s only this one test cases left to complete).
I’d like to know which test case that is (to maybe run it at the beginning or disable it). Note, this is not a matter of finding the longest running test case as that may not be the culprit. I’m explicitly looking for some way of knowing which test case is assigned to a pytest runner Python process. Calling ps shows something like python -u -c import sys; exec(eval(sys.stdin.readline())) (for as many CPU cores in the machine) which isn’t particularly helpful.
Is there a way to set the name of the test case to the process and retrieve it with system tools such as ps? I’m running those test cases on Linux, in case that’s relevant.
Answers:
Since pytest-dist 2.4, there’s a solution to showing which test case is running. It requires an additional package setproctitle.
Identifying workers from the system environment
New in version 2.4
If the setproctitle package is installed, pytest-xdist will use it to update the process title (command line) on its workers to show their current state. The titles used are [pytest-xdist running] file.py/node::id and [pytest-xdist idle], visible in standard tools like ps and top on Linux, Mac OS X and BSD systems. For Windows, please follow setproctitle’s pointer regarding the Process Explorer tool.
This is intended purely as an UX enhancement, e.g. to track down issues with long-running or CPU intensive tests. Errors in changing the title are ignored silently. Please try not to rely on the title format or title changes in external scripts.
https://pypi.org/project/pytest-xdist/#identifying-workers-from-the-system-environmentbas
Here is a way to see which test is running when pytest-xdist is processing.
Link to docs: https://docs.pytest.org/en/7.1.x/reference/reference.html#_pytest.hookspec.pytest_report_teststatus
Add the following function to your conftest.py file.
#conftest.py
def pytest_report_teststatus(report):
print(report.__dict__['nodeid'])
Example command to start pytest:
python -m pytest -n 3 -s --show-capture=no --disable-pytest-warnings
I’m executing python unit tests in parallel with pytest-forked (pytest -r sxX -n auto --cache-clear --max-worker-restart=4 --forked) and there is one test case which takes quite some time and which is running at the end while the other test case runners/CPU cores are idle (because presumably there’s only this one test cases left to complete).
I’d like to know which test case that is (to maybe run it at the beginning or disable it). Note, this is not a matter of finding the longest running test case as that may not be the culprit. I’m explicitly looking for some way of knowing which test case is assigned to a pytest runner Python process. Calling ps shows something like python -u -c import sys; exec(eval(sys.stdin.readline())) (for as many CPU cores in the machine) which isn’t particularly helpful.
Is there a way to set the name of the test case to the process and retrieve it with system tools such as ps? I’m running those test cases on Linux, in case that’s relevant.
Since pytest-dist 2.4, there’s a solution to showing which test case is running. It requires an additional package setproctitle.
Identifying workers from the system environment
New in version 2.4
If the setproctitle package is installed, pytest-xdist will use it to update the process title (command line) on its workers to show their current state. The titles used are [pytest-xdist running] file.py/node::id and [pytest-xdist idle], visible in standard tools like ps and top on Linux, Mac OS X and BSD systems. For Windows, please follow setproctitle’s pointer regarding the Process Explorer tool.
This is intended purely as an UX enhancement, e.g. to track down issues with long-running or CPU intensive tests. Errors in changing the title are ignored silently. Please try not to rely on the title format or title changes in external scripts.
https://pypi.org/project/pytest-xdist/#identifying-workers-from-the-system-environmentbas
Here is a way to see which test is running when pytest-xdist is processing.
Link to docs: https://docs.pytest.org/en/7.1.x/reference/reference.html#_pytest.hookspec.pytest_report_teststatus
Add the following function to your conftest.py file.
#conftest.py
def pytest_report_teststatus(report):
print(report.__dict__['nodeid'])
Example command to start pytest:
python -m pytest -n 3 -s --show-capture=no --disable-pytest-warnings