concurrent.futures.ProcessPoolExecutor vs multiprocessing.pool.Pool
Question:
Please explain to me what is the difference between these two classes?
I noticed multiprocessing module existed in Python 2. But functionally?
Answers:
As stated in the documentation, concurrent.futures.ProcessPoolExecutor is a wrapper around a multiprocessing.Pool. As such, the same limitations of multiprocessing apply (e.g. objects need to be pickleable).
However, concurrent.futures aims to provide an abstract interface that can be used to manage different types of asynchronous tasks in a convenient way. e.g. changing your async strategy from using process pools to using threads is frequently as simple as changing one or two lines of code (rather than needing to code it all up yourself). Another (related) benefit in the abstraction is that concurrent.futures provides a single API to remember — And you can pick the Executor that is most suited for the task. Is using your process IO bound? Awesome, use a ThreadPoolExecutor. Are you going to have trouble speeding things up because of the Global Interpreter Lock (GIL)? No problem, use a ProcessPoolExecutor.
Please explain to me what is the difference between these two classes?
I noticed multiprocessing module existed in Python 2. But functionally?
As stated in the documentation, concurrent.futures.ProcessPoolExecutor is a wrapper around a multiprocessing.Pool. As such, the same limitations of multiprocessing apply (e.g. objects need to be pickleable).
However, concurrent.futures aims to provide an abstract interface that can be used to manage different types of asynchronous tasks in a convenient way. e.g. changing your async strategy from using process pools to using threads is frequently as simple as changing one or two lines of code (rather than needing to code it all up yourself). Another (related) benefit in the abstraction is that concurrent.futures provides a single API to remember — And you can pick the Executor that is most suited for the task. Is using your process IO bound? Awesome, use a ThreadPoolExecutor. Are you going to have trouble speeding things up because of the Global Interpreter Lock (GIL)? No problem, use a ProcessPoolExecutor.