python multiprocess.Pool show results in order in stdout

Question

In multiprocessing.Pool I am trying to show my prints in the the same order.

from multiprocessing import Pool
import time
def func(arg):
    time.sleep(0.001)
    print(arg, end=" ")

proc_pool = Pool(4)
proc_pool.map(func, range(30))

The output is: 0 1 8 9 10 11 14 15 6 7 16 17 4 5 12 13 18 19 2 3 or similar. It is not in the order 0 1 2 3 ...

I know that imap can give better ordering … but it’s still not exactly what I want. I could override the print function to save it into a variable and print them all at once – but I would prefer showing them as soon as they complete – not when all complete.

Asked By: AbdealiJK

||

Source

Answer 1

Given that a Pool distributes the elements of the list amongst the worker processes (which are scheduled by the operating system), there is no way that you can guarantee internal processing order with map.

The best you can probably do (with map) is change the input into a list of tuples (sequence_number, data), have the worker function return (sequence_number, result) and than sort by sequence_number.

If you want to start processing items as soon as they are finished, use imap or imap_unordered. Using imap will preserve the same order as the input iterable. If you use the same trick with the (sequence_number, result) tuple with imap, you can save the results in a list and print them once you have a sequence without gaps.

Answered By: Roland Smith

Answer 2

Clarification – OP is seeking to get order as soon as a process is finished (and not wait for all processes to finish). Map does guarantee output order – but you need to wait for all the processes to be done.

Here is another discussion highlighting this –
Another Stack Discussion

Here is the edited code and the results differentiating between the two for folks like me who may ahve gotten confused.

from multiprocessing import Pool
import time


def func(arg):
   time.sleep(0.001)
   print("process order: "+str(arg))
   return arg
if __name__=='__main__':
   proc_pool = Pool(4)
   results = proc_pool.map(func, range(30))
   proc_pool.close()
   proc_pool.join()

   for a in results:
      print("output order"+str(a))`

The results are:

process order: 6
process order: 7
process order: 14

output order0
output order1
output order2
output order3
output order4
output order5
output order6
output order7
output order8
output order9
output order10
output order11
output order12
output order13
output order14
output order15
output order16
output order17
output order18
output order19
output order20
output order21
output order22
output order23
output order24
output order25
output order26
output order27
output order28
output order29

Answered By: pythOnometrist

python multiprocess.Pool show results in order in stdout

Question:

Answers: