Can I get a return value from multiprocessing.Process?
Question:
I’ve implemented some simple parallelism in a Monte Carlo code using the Python multiprocessing module. I have code that looks like:
montecarlos = [MonteCarlo(f,fargs) for fargs in farglist]
jobs = [multiprocessing.Process(mc) for mc in montecarlos]
for job in jobs: job.start()
for job in jobs: job.join()
results = [mc.results for mc in montecarlos]
However, when I look at the results list, it looks like the monte carlo iterators haven’t even started. I know that they have, because I can have the processes print out information during the monte carlo steps. So I’m doing something dumb. I had thought the job.join() would keep the results list from being constructed until everything had run, and thus the mc.results field would be updated.
I realize I haven’t told you the details of my monte carlo routine, and hope that it doesn’t matter, and that the mistake I’m making is in my interpretation of what multiprocessing does. Thanks in advance for any help you can offer.
Answers:
The MonteCarlo objects have been pickled and sent to child processes to be run – the .results attribute in this process isn’t populated because the local mc has never been run.
If you create a multiprocessing.Queue, you can pass that into each MonteCarlo job, and when it finishes it should put the result in there. Then the top-level can wait for values from the queue. (Under the hood this will pickle and unpickle the result object.)
result_queue = multiprocessing.Queue()
montecarlos = [MonteCarlo(result_queue, f,fargs) for fargs in farglist]
jobs = [multiprocessing.Process(mc) for mc in montecarlos]
for job in jobs: job.start()
for job in jobs: job.join()
results = [result_queue.get() for mc in montecarlos]
I’ve implemented some simple parallelism in a Monte Carlo code using the Python multiprocessing module. I have code that looks like:
montecarlos = [MonteCarlo(f,fargs) for fargs in farglist]
jobs = [multiprocessing.Process(mc) for mc in montecarlos]
for job in jobs: job.start()
for job in jobs: job.join()
results = [mc.results for mc in montecarlos]
However, when I look at the results list, it looks like the monte carlo iterators haven’t even started. I know that they have, because I can have the processes print out information during the monte carlo steps. So I’m doing something dumb. I had thought the job.join() would keep the results list from being constructed until everything had run, and thus the mc.results field would be updated.
I realize I haven’t told you the details of my monte carlo routine, and hope that it doesn’t matter, and that the mistake I’m making is in my interpretation of what multiprocessing does. Thanks in advance for any help you can offer.
The MonteCarlo objects have been pickled and sent to child processes to be run – the .results attribute in this process isn’t populated because the local mc has never been run.
If you create a multiprocessing.Queue, you can pass that into each MonteCarlo job, and when it finishes it should put the result in there. Then the top-level can wait for values from the queue. (Under the hood this will pickle and unpickle the result object.)
result_queue = multiprocessing.Queue()
montecarlos = [MonteCarlo(result_queue, f,fargs) for fargs in farglist]
jobs = [multiprocessing.Process(mc) for mc in montecarlos]
for job in jobs: job.start()
for job in jobs: job.join()
results = [result_queue.get() for mc in montecarlos]