Multiprocessing taking more time
Question:
from multiprocessing import Pool
from datetime import datetime
import time
def elapsed_time(start_time,calling_func):
delta = (time.time() - start_time)
#print("Took "+str(round(delta/60,3))+" min time for "+calling_func+" to complete")
print("Took "+str(round(delta/60,3))+" mins "+calling_func)
l =[]
squared = []
squared1 = []
for i in range(10):
l.append(i)
################### TRADITIONAL ###############
def square():
start_time = time.time()
for i in range(len(l)):
squared.append(i*i)
elapsed_time(start_time,"to SQUARE using traditional approach")
l.append(i)
################### TRADITIONAL ###############
square()
################### MULTIPROCESSING ###############
def row_logic(row):
# print("List Item:::", row)
result = row*row
# squared.append(row*row)
# print("Processed row:::", row)
return result
# def main():
# start_time = time.time()
# pool = Pool(3)
# # for result in pool.map(row_logic, l):
# # print(result)
# # pool.map(row_logic, l)
# for result in pool.map(row_logic, l):
# squared1.append(result)
# elapsed_time(start_time,"to SQUARE using POOL")
if __name__ == '__main__':
# freeze_support() here if program needs to be frozen
# main() #
start_time = time.time()
pool = Pool(3)
for result in pool.map(row_logic, l):
squared1.append(result)
elapsed_time(start_time,"to SQUARE using multiprocessing approach")
################### MULTIPROCESSING ###############
print("SQUARED LIST using traditional approach:::",squared)
print("SQUARED LIST using multiprocessing approach:::",squared1)
Output:
PS C:UsersrrsolomoDownloadspython_practice> & C:/Users/rrsolomo/anaconda3/python.exe c:/Users/rrsolomo/Downloads/python_practice/zzz_TEST_3.py
Took 0.0 mins to SQUARE using traditional approach
Took 0.0 mins to SQUARE using traditional approach
SQUARED LIST using traditional approach::: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
SQUARED LIST using multiprocessing approach::: []
Took 0.0 mins to SQUARE using traditional approach
SQUARED LIST using traditional approach::: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
SQUARED LIST using multiprocessing approach::: []
Took 0.0 mins to SQUARE using traditional approach
SQUARED LIST using traditional approach::: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
SQUARED LIST using multiprocessing approach::: []
Took 0.002 mins to SQUARE using multiprocessing approach
SQUARED LIST using traditional approach::: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
SQUARED LIST using multiprocessing approach::: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 81]
I have the following questions:
- Why does it take more time to process the function using Multiprocessing? 0.002 mins vs 0.0 mins
- Trying to understand why we’re seeing multiple print lines from elapsed()
Please correct me if my understanding is not right.
Note: Am using Windows machine to run this code
Since this seems to be a simpler task, I used the below code as well:
from multiprocessing import Pool
from datetime import datetime
import time
import math
def elapsed_time(start_time,calling_func):
delta = (time.time() - start_time)
#print("Took "+str(round(delta/60,3))+" min time for "+calling_func+" to complete")
print("Took "+str(round(delta/60,3))+" mins "+calling_func)
N = 5000000
serial_result = []
def cube(x):
return math.sqrt(x)
def serial_cube():
start_time = time.time()
for i in range(N):
serial_result.append(math.sqrt(i))
elapsed_time(start_time,"to CUBE using SERIAL method")
serial_cube()
if __name__ == "__main__":
start_time = time.time()
with Pool() as pool:
result = pool.map(cube, range(10,N))
elapsed_time(start_time,"to CUBE using POOL")
# print(type(result))```
And it printed:
Took 0.012 mins to CUBE using SERIAL method
Took 0.031 mins to CUBE using SERIAL method
Took 0.031 mins to CUBE using SERIAL method
Took 0.032 mins to CUBE using SERIAL method
Took 0.032 mins to CUBE using SERIAL method
Took 0.032 mins to CUBE using SERIAL method
Took 0.032 mins to CUBE using SERIAL method
Took 0.032 mins to CUBE using SERIAL method
Took 0.032 mins to CUBE using SERIAL method
Took 0.047 mins to CUBE using POOL
I am trying to understand the functionality of MultiProcessing.
Answers:
Context switching is the cause of this.
https://en.wikipedia.org/wiki/Context_switch
By big operation he means that the function itself takes a long time.
Your code is just doing the square root calculation with the Pool.
root calculation is simple function. so multi processing is takes more time.
read file -> split -> sort -> counting start alphabet…
in this case, multi process is faster.
because read file takes IO time, that means, serial task have idle cpu time.
You aren’t appreciating how fast something like row*row is, and how much overhead multiprocessing requires. Think of the serialization/deserialization to send the results using some IPC approach. Compared to math.sqrt(row) even, that’s a lot. There has to be more appreciable work.
I found around where multiprocessing starts winning on my machine. Note, I simplified your code and fixed one big error which is that you were running all your serial profiling code in the subprocess before actually getting to the part you were interested int. This is because your timing code was at the module level so each time a new process was started the serial profiling code would be run. This is why it is printed out several times.
Anyway, here is an example:
(py311) jarrivillaga-mbp16-2019:~ jarrivillaga$ cat mptest.py
Gives:
from multiprocessing import Pool
from datetime import datetime
import time
import datetime
import math
def elapsed_time(start_time, msg):
delta = datetime.timedelta(seconds=time.perf_counter() - start_time)
print(f"Took {delta} {msg}")
def func(row):
return math.sin(row)*math.cos(row)/(math.tan(row) or 1)
if __name__ == '__main__':
l = list(range(10_000_000))
start_time = time.perf_counter()
result = [func(row) for row in l]
elapsed_time(start_time,"to SQUARE using traditional approach")
start_time = time.perf_counter()
pool = Pool(2)
squared1 = pool.map(func, l)
elapsed_time(start_time,"to SQUARE using multiprocessing approach")
Now, running it:
(py311) jarrivillaga-mbp16-2019:~ jarrivillaga$ python mptest.py
Took 0:00:03.091847 to SQUARE using traditional approach
Took 0:00:02.686483 to SQUARE using multiprocessing approach
So here multiprocessing is just barely winning.
Note, if I switch back to just def func(row): return row*row, look what happens:
(py311) jarrivillaga-mbp16-2019:~ jarrivillaga$ python mptest.py
Took 0:00:00.802870 to SQUARE using traditional approach
Took 0:00:01.792571 to SQUARE using multiprocessing approach
Now, what if I do the simplest operation possible, def func(row): pass:
(py311) jarrivillaga-mbp16-2019:~ jarrivillaga$ python mptest.py
Took 0:00:00.566595 to SQUARE using traditional approach
Took 0:00:01.099248 to SQUARE using multiprocessing approach
So, aspect of the overhead that both approaches share is that they both build a list serially (that’s what pool.map does for you). So, you need to cal a function, which costs time, and append the result to a list 10,000,000. These are variable costs that scale with the size of your output. But on top of that, multiprocessing actually has to serialize and deserialize the results (using pickle basically), plus send the information between processes using some IPC approach. These also are variable costs that scale with each unit of work. And you can’t forget the costs of actually starting the processes, (although this is fixed and up-front).
What if I add more work?
def func(row):
temp = math.sin(row)*math.cos(row)/(math.tan(row) or 1)
return temp + math.log(row or 1, 42) * math.log(row or 1)
Then multiprocessing seems to be doing even better relatively, since the work per unit is finally appreciable enough to make up for the fixed and variable costs associated with multiprocessing:
(py311) jarrivillaga-mbp16-2019:~ jarrivillaga$ python mptest.py
Took 0:00:05.441318 to SQUARE using traditional approach
Took 0:00:03.616397 to SQUARE using multiprocessing approach
You can also play around with various things here, the size of the inputs, l, the number of processes (I used 2, try it with 3 or 4).
from multiprocessing import Pool
from datetime import datetime
import time
def elapsed_time(start_time,calling_func):
delta = (time.time() - start_time)
#print("Took "+str(round(delta/60,3))+" min time for "+calling_func+" to complete")
print("Took "+str(round(delta/60,3))+" mins "+calling_func)
l =[]
squared = []
squared1 = []
for i in range(10):
l.append(i)
################### TRADITIONAL ###############
def square():
start_time = time.time()
for i in range(len(l)):
squared.append(i*i)
elapsed_time(start_time,"to SQUARE using traditional approach")
l.append(i)
################### TRADITIONAL ###############
square()
################### MULTIPROCESSING ###############
def row_logic(row):
# print("List Item:::", row)
result = row*row
# squared.append(row*row)
# print("Processed row:::", row)
return result
# def main():
# start_time = time.time()
# pool = Pool(3)
# # for result in pool.map(row_logic, l):
# # print(result)
# # pool.map(row_logic, l)
# for result in pool.map(row_logic, l):
# squared1.append(result)
# elapsed_time(start_time,"to SQUARE using POOL")
if __name__ == '__main__':
# freeze_support() here if program needs to be frozen
# main() #
start_time = time.time()
pool = Pool(3)
for result in pool.map(row_logic, l):
squared1.append(result)
elapsed_time(start_time,"to SQUARE using multiprocessing approach")
################### MULTIPROCESSING ###############
print("SQUARED LIST using traditional approach:::",squared)
print("SQUARED LIST using multiprocessing approach:::",squared1)
Output:
PS C:UsersrrsolomoDownloadspython_practice> & C:/Users/rrsolomo/anaconda3/python.exe c:/Users/rrsolomo/Downloads/python_practice/zzz_TEST_3.py
Took 0.0 mins to SQUARE using traditional approach
Took 0.0 mins to SQUARE using traditional approach
SQUARED LIST using traditional approach::: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
SQUARED LIST using multiprocessing approach::: []
Took 0.0 mins to SQUARE using traditional approach
SQUARED LIST using traditional approach::: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
SQUARED LIST using multiprocessing approach::: []
Took 0.0 mins to SQUARE using traditional approach
SQUARED LIST using traditional approach::: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
SQUARED LIST using multiprocessing approach::: []
Took 0.002 mins to SQUARE using multiprocessing approach
SQUARED LIST using traditional approach::: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
SQUARED LIST using multiprocessing approach::: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 81]
I have the following questions:
- Why does it take more time to process the function using Multiprocessing? 0.002 mins vs 0.0 mins
- Trying to understand why we’re seeing multiple print lines from elapsed()
Please correct me if my understanding is not right.
Note: Am using Windows machine to run this code
Since this seems to be a simpler task, I used the below code as well:
from multiprocessing import Pool
from datetime import datetime
import time
import math
def elapsed_time(start_time,calling_func):
delta = (time.time() - start_time)
#print("Took "+str(round(delta/60,3))+" min time for "+calling_func+" to complete")
print("Took "+str(round(delta/60,3))+" mins "+calling_func)
N = 5000000
serial_result = []
def cube(x):
return math.sqrt(x)
def serial_cube():
start_time = time.time()
for i in range(N):
serial_result.append(math.sqrt(i))
elapsed_time(start_time,"to CUBE using SERIAL method")
serial_cube()
if __name__ == "__main__":
start_time = time.time()
with Pool() as pool:
result = pool.map(cube, range(10,N))
elapsed_time(start_time,"to CUBE using POOL")
# print(type(result))```
And it printed:
Took 0.012 mins to CUBE using SERIAL method
Took 0.031 mins to CUBE using SERIAL method
Took 0.031 mins to CUBE using SERIAL method
Took 0.032 mins to CUBE using SERIAL method
Took 0.032 mins to CUBE using SERIAL method
Took 0.032 mins to CUBE using SERIAL method
Took 0.032 mins to CUBE using SERIAL method
Took 0.032 mins to CUBE using SERIAL method
Took 0.032 mins to CUBE using SERIAL method
Took 0.047 mins to CUBE using POOL
I am trying to understand the functionality of MultiProcessing.
Context switching is the cause of this.
https://en.wikipedia.org/wiki/Context_switch
By big operation he means that the function itself takes a long time.
Your code is just doing the square root calculation with the Pool.
root calculation is simple function. so multi processing is takes more time.
read file -> split -> sort -> counting start alphabet…
in this case, multi process is faster.
because read file takes IO time, that means, serial task have idle cpu time.
You aren’t appreciating how fast something like row*row is, and how much overhead multiprocessing requires. Think of the serialization/deserialization to send the results using some IPC approach. Compared to math.sqrt(row) even, that’s a lot. There has to be more appreciable work.
I found around where multiprocessing starts winning on my machine. Note, I simplified your code and fixed one big error which is that you were running all your serial profiling code in the subprocess before actually getting to the part you were interested int. This is because your timing code was at the module level so each time a new process was started the serial profiling code would be run. This is why it is printed out several times.
Anyway, here is an example:
(py311) jarrivillaga-mbp16-2019:~ jarrivillaga$ cat mptest.py
Gives:
from multiprocessing import Pool
from datetime import datetime
import time
import datetime
import math
def elapsed_time(start_time, msg):
delta = datetime.timedelta(seconds=time.perf_counter() - start_time)
print(f"Took {delta} {msg}")
def func(row):
return math.sin(row)*math.cos(row)/(math.tan(row) or 1)
if __name__ == '__main__':
l = list(range(10_000_000))
start_time = time.perf_counter()
result = [func(row) for row in l]
elapsed_time(start_time,"to SQUARE using traditional approach")
start_time = time.perf_counter()
pool = Pool(2)
squared1 = pool.map(func, l)
elapsed_time(start_time,"to SQUARE using multiprocessing approach")
Now, running it:
(py311) jarrivillaga-mbp16-2019:~ jarrivillaga$ python mptest.py
Took 0:00:03.091847 to SQUARE using traditional approach
Took 0:00:02.686483 to SQUARE using multiprocessing approach
So here multiprocessing is just barely winning.
Note, if I switch back to just def func(row): return row*row, look what happens:
(py311) jarrivillaga-mbp16-2019:~ jarrivillaga$ python mptest.py
Took 0:00:00.802870 to SQUARE using traditional approach
Took 0:00:01.792571 to SQUARE using multiprocessing approach
Now, what if I do the simplest operation possible, def func(row): pass:
(py311) jarrivillaga-mbp16-2019:~ jarrivillaga$ python mptest.py
Took 0:00:00.566595 to SQUARE using traditional approach
Took 0:00:01.099248 to SQUARE using multiprocessing approach
So, aspect of the overhead that both approaches share is that they both build a list serially (that’s what pool.map does for you). So, you need to cal a function, which costs time, and append the result to a list 10,000,000. These are variable costs that scale with the size of your output. But on top of that, multiprocessing actually has to serialize and deserialize the results (using pickle basically), plus send the information between processes using some IPC approach. These also are variable costs that scale with each unit of work. And you can’t forget the costs of actually starting the processes, (although this is fixed and up-front).
What if I add more work?
def func(row):
temp = math.sin(row)*math.cos(row)/(math.tan(row) or 1)
return temp + math.log(row or 1, 42) * math.log(row or 1)
Then multiprocessing seems to be doing even better relatively, since the work per unit is finally appreciable enough to make up for the fixed and variable costs associated with multiprocessing:
(py311) jarrivillaga-mbp16-2019:~ jarrivillaga$ python mptest.py
Took 0:00:05.441318 to SQUARE using traditional approach
Took 0:00:03.616397 to SQUARE using multiprocessing approach
You can also play around with various things here, the size of the inputs, l, the number of processes (I used 2, try it with 3 or 4).