Run a method in parallel inside another method in python

Question

I am trying and failing to run the following computation in python. I have a class, let’s call it class Telescope, which has some method, compute_something(self, args). Inside the same class I have a method, simulation, which needs to call compute_something eight times with different arguments. Each of these runs is independent from each other, so I could just run them at the same time, in order to speed up time. Does anyone know how to do this?

I have multiprocessing the same way I’ve done it outside of classes but I have not been sucessful. I have read that some people would define the method outside the class, but this is inviable in this case.

Asked By: Bohm Arahnmob

||

Source

Answer 1

What you need is the "Pool" class from the multiprocessing library

Here’s some example code:

import multiprocessing as mp

class Telescope:
    def compute_something(self, a, b, c):
        return a * b * c #Do your computations here
    def simulation(self):
        #The input to starmap needs to be formatted like this: [[arg1 for first call, arg2 for first call, arg3 for first call], [arg1 for second call, arg2 for second call, arg3 for second call]]
        starmap_in = []
        #Populate starmap_in
        for i in range(5):
            starmap_in.append(list(range(i, i+3)))

        print(starmap_in)

        with mp.Pool(5) as pool:
            print(pool.starmap(self.compute_something, starmap_in)) #This is what calls the function 5 times in paralell
            pool.close()

x = Telescope()

x.simulation()

However if you will be using the self variable in your function, doing it like this is a really bad idea. You need locks to do this if you want to access self.

import multiprocessing as mp

def init_pool_processes(the_lock):
        #Initialize each process with a global variable lock.
        
        global lock
        lock = the_lock

class Telescope:
    def __init__(self, num):
        self.num = num
    def compute_something(self, a, b, c):
        lock.acquire()
        res = self.num * a * b * c #Do your computations here
        lock.release()
        return res
    def simulation(self):

        lock = mp.Lock()

        #The input to starmap needs to be formatted like this: [[arg1 for first call, arg2 for first call, arg3 for first call], [arg1 for second call, arg2 for second call, arg3 for second call]]
        starmap_in = []
        #Populate starmap_in
        for i in range(5):
            starmap_in.append(list(range(i, i+3)))

        with mp.Pool(5, initializer=init_pool_processes, initargs=(lock,)) as pool:
            print(pool.starmap(self.compute_something, starmap_in)) #This is what calls the function 5 times in paralell
            pool.close()

x = Telescope(10)

x.simulation()

Answered By: drake14w

Run a method in parallel inside another method in python

Question:

Answers: