Return variable from python multi-process

Question

My original code looks similar to this toy example. It creates two random matrices, multiplies them and saves the output in a list.

import numpy as np

n = 100
N = 10
out = [1] * N

for i in range(0, N):

    A = np.random.rand(n, n)
    B = np.random.rand(n, n)

    out[i] = A * B

Now I am trying to parallelize this loop using multiprocessing. I define the function g and run_complex_operations in defs.py

import numpy as np
    
def g(n):
    A = np.random.rand(n, n)
    B = np.random.rand(n, n)

    C = A * B


def run_complex_operations(operation, input, pool):
    pool.map(operation, input)
    print(input)

and parallelize the code as follows:

import defs   
import numpy as np
import multiprocessing as mp

n = 100
N = 10
l = [n] * N

if __name__ == '__main__':
    processes_pool = mp.Pool(N)
    defs.run_complex_operations(defs.g, range(N), processes_pool)

My question is: How can I save the resulting matrix C in each iteration in the list out, so that I end up with a list of 10 matrices?

Update:

Solved.

Asked By: eigenvector

||

Source

Answer 1

the keyword return from python functions tutorial in python returns its parameter back to the caller, you can use it to return the result of the function back to out

import numpy as np
import multiprocessing as mp

def g(n):
    A = np.random.rand(n, n)
    B = np.random.rand(n, n)

    return A * B

def run_complex_operations(operation, input, pool):
    result = pool.map(operation, input)
    print(input)
    return result

n = 100
N = 10
l = [n] * N

if __name__ == '__main__':
    processes_pool = mp.Pool(N)
    out = run_complex_operations(g, range(N), processes_pool)
    print(len(out))  # 10

Edit: a small trick, if you instead want to fill l variable with the output your can use inplace assignment of lists as follows.

l[:] = run_complex_operations(g, range(N), processes_pool)

which will take the returns of the function and put each element returned to an index in the list, …. but predefining l has no real benefit over l = run_complex_operations...

small note, you are doing an element-wise multiplication, not matrix multiplication, you should use A @ B for matrix multiplication.

Answered By: Ahmed AEK

Return variable from python multi-process

Question:

Answers: