Accessing shared resource across master and worker nodes using pytest-xdist

Question

I’m trying to share a list of random entries from a database (my definition of a shared resource) across master and worker nodes and parallelize tests using pytest-xdist. My code follows the structure below:

## in conftest.py
def get_db_entries():
   connect to db
   run a query with group by and random
   returns one random entry per group as a list of dictionaries.

I adapted the suggestion provided by https://hackebrot.github.io/pytest-tricks/shared_directory_xdist/ to share the db entries across master and worker nodes:

# in conftest.py
def pytest_configure(config):
    if is_master(config):
        # share db_entries across master and worker nodes.
        config.db_samples = get_db_entries()

def pytest_configure_node(node):
    """xdist hook"""
    node.slaveinput['db_entries'] = node.config.db_samples

def is_master(config):
    """True if the code running the given pytest.config object is running in a xdist master
    node or not running xdist at all.
    """
    return not hasattr(config, 'slaveinput')

@pytest.fixture
def db_samples(request):
    """Returns a unique and temporary directory which can be shared by
    master or worker nodes in xdist runs.
    """
    if is_master(request.config):
        return request.config.db_samples
    else:
        return request.config.slaveinput['db_entries']

I’m able to share these db entries across master and workers using the above approach as follows:

# in test-db.py
def test_db(db_samples):
    for sample in samples:
       # test each sample.

Until now, there is no parallelization of test cases. I’m just sharing the same database entries across master and worker nodes.

My question: How do I parameterize the shared resource state (database entries) so that I can use pytest-xdist to execute these tests in parallel?

I would like to do something along the lines of:

# in conftest.py
samples = db_samples() # will FAIL coz fixture can't be invoked directly.

@pytest.fixture(params=samples)
def get_sample(request):
   return request.param

# in test-db.py
def test_db(get_sample):
    # test one sample.

Asked By: Bhushan

||

Source

Answer 1

Thanks to the suggestion from @hoefling, I was able to use pytest-xdist hooks and pytest_generate_tests to get a working solution.

# in conftest.py
def pytest_configure(config):
    if is_master(config):
        # share db_entries across master and worker nodes.
        config.db_samples = get_db_entries()

def pytest_configure_node(node):
    """xdist hook"""
    node.slaveinput['db_entries'] = node.config.db_samples

def is_master(config):
    """True if the code running the given pytest.config object is running in a xdist master
    node or not running xdist at all.
    """
    return not hasattr(config, 'slaveinput')

def pytest_generate_tests(metafunc):
    if 'sample' in metafunc.fixturenames:
        if is_master(metafunc.config):
            samples = metafunc.config.db_samples
        else:
            samples = metafunc.config.slaveinput['db_entries']
        metafunc.parametrize("sample", samples)

@pytest.fixture
def sample(request):
    return request.param

# in test-db.py
def test_db(sample):
    # test one sample.

@hoefling suggestion to use pytest_generate_tests hook was the missing piece. Using this hook to parametrize the test fixture helped in solving this problem.

Answered By: Bhushan

Answer 2

Also worthwhile to check out this thread, which allows for a customizable teardown function as well:

https://github.com/pytest-dev/pytest-xdist/issues/271#issuecomment-826396320

Answered By: Justin Furuness

Accessing shared resource across master and worker nodes using pytest-xdist

Question:

Answers: