Run Pytest Classes in Custom Order

Question

I am writing tests with pytest in pycharm. The tests are divided into various classes.
I would like to specify certain classes that have to run before other classes.
I have seen various questions on stackoverflow (such as specifying pytest tests to run from a file and how to run a method before all other tests).
These and various others questions wanted to choose specific functions to run in order. This can be done, I understand, using fixtures or with pytest ordering.
I don’t care which functions from each class run first. All I care about is that the classes run in the order I specify. Is this possible?

Asked By: user613

||

Source

Answer 1

Approach

You can use the pytest_collection_modifyitems hook to modify the order of collected tests (items) in place. This has the additional benefit of not having to install any third party libraries.

With some custom logic, this allows to sort by class.

Full example

Say we have three test classes:

TestExtract
TestTransform
TestLoad

Say also that, by default, the test order of execution would be alphabetical, i.e.:

TestExtract -> TestLoad -> TestTransform

which does not work for us due to test class interdependencies.

We can add pytest_collection_modifyitems to conftest.py as follows to enforce our desired execution order:

# conftest.py
def pytest_collection_modifyitems(items):
    """Modifies test items in place to ensure test classes run in a given order."""
    CLASS_ORDER = ["TestExtract", "TestTransform", "TestLoad"]
    class_mapping = {item: item.cls.__name__ for item in items}

    sorted_items = items.copy()
    # Iteratively move tests of each class to the end of the test queue
    for class_ in CLASS_ORDER:
        sorted_items = [it for it in sorted_items if class_mapping[it] != class_] + [
            it for it in sorted_items if class_mapping[it] == class_
        ]
    items[:] = sorted_items

Some comments on the implementation details:

Test classes can live in different modules
CLASS_ORDER does not have to be exhaustive. You can reorder just those classes on which to want to enforce an order (but note: if reordered, any non-reordered class will execute before any reordered class)
The test order within the classes is kept unchanged
It is assumed that test classes have unique names
items must be modified in place, hence the final items[:] assignment

Answered By: swimmer

Answer 2

@swimmer answer is great. I’ve modified it slightly to work with test organized into functions instead of classes.

def pytest_collection_modifyitems(session, config, items):
    """Modifies test items in place to ensure test functions run in a given order"""
    function_order = ["test_one", "test_two"]
    # OR
    # function_order = ["test_one[1]", "test_two[2]"]  
    function_mapping = {item: item.name.split("[")[0] 
                        if "]" not in function_order[0] 
                        else item.name
                        for item in items}

    sorted_items = items.copy()
    for func_ in function_order:
        sorted_items = [it for it in sorted_items if function_mapping[it] != func_] + [it for it in sorted_items if function_mapping[it] == func_]
    items[:] = sorted_items

Answered By: david-engelmann

Run Pytest Classes in Custom Order

Question:

Answers:

Approach

Full example