Python – Multiprocessing of multiple variable length iterators

Question

I am having trouble identifying how to iterate over two unequal length lists in Python using multiprocessing. I basically want each value in each list to be computed against each other which I have been accomplishing in a nested loop. What I am finding is that the routine runs through each list sequentially and then terminates after the shortest list has been iterated through (this does make sense). I have to assume there is a way to get this to work. Thx in advance for any guidance.

def run_test(short, long):
    print(short)
    print(long)

if __name__ == '__main__':
    short_value = [1,2,3]
    long_value = [4,5,6,7,8,9]
    args = [short_value, long_value]
    all_data=zip(short_value,long_value)
    data=list(all_data)
    with Pool(1) as p:
        p.starmap(run_test, zip(short_value, long_value))
        p.close()
        p.join()

1 4 2 5 3 6

(never sees 7, 8, and 9 in the longer list)

Asked By: Sumobull

||

Source

Answer 1

If you want to compute each value in one list against each value in another list, you’ll need to compute the Cartesian product of the two lists. You can use itertools.product to generate all possible pairs, and then pass these pairs to the run_test function using multiprocessing. Following is the modified code:

from itertools import product
from multiprocessing import Pool

def run_test(short, long):
    print(f'{short} {long}') # Printed on the same line for clarity

if __name__ == '__main__':
    short_value = [1, 2, 3]
    long_value = [4, 5, 6, 7, 8, 9]

    # Generate all possible pairs (Cartesian product) from the two lists
    all_pairs = list(product(short_value, long_value))

    with Pool(1) as p:
        p.starmap(run_test, all_pairs)
        p.close()
        p.join()

Now, the code will iterate over all possible pairs of values from both lists, and the output will include pairs with values 7, 8, and 9 from the larger list.

Answered By: Bilesh Ganguly

Answer 2

Itertools.product() works.

itertools.product() 

def run_test(args):
    print(args)
    
if __name__ == '__main__':
    
    paramlist = list(itertools.product([1,2,3], [4,5,6,7,8,9]))
    with Pool(1) as p:
        p.map(run_test, paramlist)
        p.close()
        p.join()

Results:

(1, 4) (1, 5) (1, 6) (1, 7) (1, 8) (1, 9) (2, 4) (2, 5) (2, 6) (2, 7) (2, 8) (2, 9) (3, 4) (3, 5) (3, 6) (3, 7) (3, 8) (3, 9)

Answered By: Sumobull

Python – Multiprocessing of multiple variable length iterators

Question:

Answers: