What is the difference between multiprocessing and subprocess?
Question:
My work should use parallel techniques, and I a new user of python. So I wonder if you could share some material about the python multiprocessing and subprocess modules. What is the difference between these two?
Answers:
If you want to call an external program (especially one not written in Python) use subprocess.
If you want to call a Python function in a subprocess, use multiprocessing.
(If the program is written in Python, but is also importable, then I would try to call its functions using multiprocessing, rather than calling it externally through subprocess.)
Subprocess spawns new processes, but aside from stdin/stdout and whatever other APIs the other program may implement you have no means to communicate with them. Its main purpose is to launch processes that are completely separate from your own program.
Multiprocessing also spawns new processes, but they run your code, and are designed to communicate with each other. You use it to divide tasks within your own program across multiple CPU cores.
The subprocess module lets you run and control other programs. Anything you can start with the command line on the computer, can be run and controlled with this module. Use this to integrate external programs into your Python code.
The multiprocessing module lets you divide tasks written in python over multiple processes to help improve performance. It provides an API very similar to the threading module; it provides methods to share data across the processes it creates, and makes the task of managing multiple processes to run Python code (much) easier. In other words, multiprocessing lets you take advantage of multiple processes to get your tasks done faster by executing code in parallel.
My work should use parallel techniques, and I a new user of python. So I wonder if you could share some material about the python multiprocessing and subprocess modules. What is the difference between these two?
If you want to call an external program (especially one not written in Python) use subprocess.
If you want to call a Python function in a subprocess, use multiprocessing.
(If the program is written in Python, but is also importable, then I would try to call its functions using multiprocessing, rather than calling it externally through subprocess.)
Subprocess spawns new processes, but aside from stdin/stdout and whatever other APIs the other program may implement you have no means to communicate with them. Its main purpose is to launch processes that are completely separate from your own program.
Multiprocessing also spawns new processes, but they run your code, and are designed to communicate with each other. You use it to divide tasks within your own program across multiple CPU cores.
The subprocess module lets you run and control other programs. Anything you can start with the command line on the computer, can be run and controlled with this module. Use this to integrate external programs into your Python code.
The multiprocessing module lets you divide tasks written in python over multiple processes to help improve performance. It provides an API very similar to the threading module; it provides methods to share data across the processes it creates, and makes the task of managing multiple processes to run Python code (much) easier. In other words, multiprocessing lets you take advantage of multiple processes to get your tasks done faster by executing code in parallel.