Python Multiprocessing: What's the difference between map and imap?
Question:
I’m trying to learn how to use Python’s multiprocessing package, but I don’t understand the difference between map and imap.
Is the difference that map returns, say, an actual array or set, while imap returns an iterator over an array or set? When would I use one over the other?
Also, I don’t understand what the chunksize argument is. Is this the number of values that are passed to each process?
Answers:
That is the difference. One reason why you might use imap instead of map is if you wanted to start processing the first few results without waiting for the rest to be calculated. map waits for every result before returning.
As for chunksize, it is sometimes more efficient to dole out work in larger quantities because every time the worker requests more work, there is IPC and synchronization overhead.
imap is from itertools module which is used for fast and memory efficiency in python.Map will return the list where as imap returns the object which generates the values for each iterations(In python 2.7).The below code blocks will clear the difference.
Map returns the list can be printed directly
from itertools import *
from math import *
integers = [1,2,3,4,5]
sqr_ints = map(sqrt, integers)
print (sqr_ints)
imap returns object which is converted to list and printed.
from itertools import *
from math import *
integers = [1,2,3,4,5]
sqr_ints = imap(sqrt, integers)
print list(sqr_ints)
Chunksize will make the iterable to be split into pieces of specified size(approximate) and each piece is submitted as a separate task.
With imap, forked calls are done in parallel, not one after another sequentially.
For example, below you’re hitting say three exchanges to get order books.
Instead of hitting exchange 1, then exchange 2, then exchange 3 sequentially, imap.pool calls are non-blocking and goes straight to all three exchanges to fetch order books as soon as you call.
from pathos.multiprocessing import ProcessingPool as Pool
pool = Pool().imap
self.pool(self.getOrderBook, Exchanges, Tickers)
I’m trying to learn how to use Python’s multiprocessing package, but I don’t understand the difference between map and imap.
Is the difference that map returns, say, an actual array or set, while imap returns an iterator over an array or set? When would I use one over the other?
Also, I don’t understand what the chunksize argument is. Is this the number of values that are passed to each process?
That is the difference. One reason why you might use imap instead of map is if you wanted to start processing the first few results without waiting for the rest to be calculated. map waits for every result before returning.
As for chunksize, it is sometimes more efficient to dole out work in larger quantities because every time the worker requests more work, there is IPC and synchronization overhead.
imap is from itertools module which is used for fast and memory efficiency in python.Map will return the list where as imap returns the object which generates the values for each iterations(In python 2.7).The below code blocks will clear the difference.
Map returns the list can be printed directly
from itertools import *
from math import *
integers = [1,2,3,4,5]
sqr_ints = map(sqrt, integers)
print (sqr_ints)
imap returns object which is converted to list and printed.
from itertools import *
from math import *
integers = [1,2,3,4,5]
sqr_ints = imap(sqrt, integers)
print list(sqr_ints)
Chunksize will make the iterable to be split into pieces of specified size(approximate) and each piece is submitted as a separate task.
With imap, forked calls are done in parallel, not one after another sequentially.
For example, below you’re hitting say three exchanges to get order books.
Instead of hitting exchange 1, then exchange 2, then exchange 3 sequentially, imap.pool calls are non-blocking and goes straight to all three exchanges to fetch order books as soon as you call.
from pathos.multiprocessing import ProcessingPool as Pool
pool = Pool().imap
self.pool(self.getOrderBook, Exchanges, Tickers)