Appending to the same list from different processes using multiprocessing

Question

I need to append objects to one list L from different processes using multiprocessing , but it returns empty list.
How can I let many processes append to list L using multiprocessing?

#!/usr/bin/python
from multiprocessing import Process

L=[]
def dothing(i,j):
    L.append("anything")
    print i

if __name__ == "__main__":
    processes=[]
    for i in range(5):
        p=Process(target=dothing,args=(i,None))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()

print L

Asked By: Adam

||

Source

Answer 1

Global variables are not shared between processes.

You need to use multiprocessing.Manager.list:

from multiprocessing import Process, Manager

def dothing(L, i):  # the managed list `L` passed explicitly.
    L.append("anything")

if __name__ == "__main__":
    with Manager() as manager:
        L = manager.list()  # <-- can be shared between processes.
        processes = []
        for i in range(5):
            p = Process(target=dothing, args=(L,i))  # Passing the list
            p.start()
            processes.append(p)
        for p in processes:
            p.join()
        print L

See Sharing state between processes¶ (Server process part).

Answered By: falsetru

Answer 2

Thanks to @falsetru for suggesting the exact documentation and providing the good code. I need to keep the order for my application and by modifying the @falsetru code, now the below code preserves the order of adding items to the list.

The sleep is helpful to catch the bugs otherwise it is hard to catch the problem with ordering of the list.

from multiprocessing import Process, Manager
from time import sleep

def dothing(L, i):  # the managed list `L` passed explicitly.
    L[i]= i
    sleep(4)

if __name__ == "__main__":
    with Manager() as manager:
        L = manager.list(range(50))  # <-- can be shared between processes.
        processes = []
        for i in range(50):
            p = Process(target=dothing, args=(L,i))  # Passing the list
            p.start()
            processes.append(p)
        for p in processes:
            p.join()
        print(L)

Answered By: eSadr

Answer 3

Falsetru’s answer worked.

But still, the list was not accessible beyond the with Manager() as manager: two changes were needed:

adding L = [] in front of the if __name__ == "__main__": statement. Must be added as for some reason the last print(L) (the one outside of if) is executed Processes + 1 times. This returns an error that L is not defined and the code breaks.
adding L = list(L)after the p.join() statement. This step is needed to change Manager.list to regular Python.list – otherwise calls to the manager.list return errors that object not readable.

from multiprocessing import Process, Manager

def dothing(L, i):  # the managed list `L` passed explicitly.
    for j in range(5):
        text = "Process " + str(i) + ", Element " + str(j)
        L.append(text)

L = [] 

if __name__ == "__main__":
    with Manager() as manager:
        L = manager.list()  # <-- can be shared between processes.
        processes = []

        for i in range(5):
            p = Process(target=dothing, args=(L,i,))  # Passing the list
            p.start()
            processes.append(p)

        for p in processes:
            p.join()

        L = list(L) 
        print("Within WITH")
        print(L)

    print("Within IF")
    print(L)

print("Outside of IF")
print(L)

Output:

Outside of IF
[]
Outside of IF
[]
Outside of IF
[]
Outside of IF
[]
Outside of IF
[]
Outside of IF
[]
Within WITH
['Process 2, Element 0','Process 2, Element 1', 'Process 2, Element 2',
'Process 2, Element 3', 'Process 2, Element 4', 'Process 1, Element 0', 
'Process 1, Element 1', 'Process 1, Element 2', 'Process 1, Element 3', 
'Process 1, Element 4', 'Process 0, Element 0', 'Process 0, Element 1', 
'Process 0, Element 2', 'Process 0, Element 3', 'Process 0, Element 4', 
'Process 4, Element 0', 'Process 4, Element 1', 'Process 4, Element 2', 
'Process 4, Element 3', 'Process 4, Element 4', 'Process 3, Element 0', 
'Process 3, Element 1', 'Process 3, Element 2', 'Process 3, Element 3', 
'Process 3, Element 4']

Within IF
['Process 2, Element 0','Process 2, Element 1', 'Process 2, Element 2', 
'Process 2, Element 3', 'Process 2, Element 4', 'Process 1, Element 0', 
'Process 1, Element 1', 'Process 1, Element 2', 'Process 1, Element 3', 
'Process 1, Element 4', 'Process 0, Element 0', 'Process 0, Element 1', 
'Process 0, Element 2', 'Process 0, Element 3', 'Process 0, Element 4', 
'Process 4, Element 0', 'Process 4, Element 1', 'Process 4, Element 2',
'Process 4, Element 3', 'Process 4, Element 4', 'Process 3, Element 0', 
'Process 3, Element 1', 'Process 3, Element 2', 'Process 3, Element 3', 
'Process 3, Element 4']

Outside of IF
['Process 2, Element 0','Process 2, Element 1', 'Process 2, Element 2', 
'Process 2, Element 3', 'Process 2, Element 4', 'Process 1, Element 0', 
'Process 1, Element 1', 'Process 1, Element 2', 'Process 1, Element 3', 
'Process 1, Element 4', 'Process 0, Element 0', 'Process 0, Element 1', 
'Process 0, Element 2', 'Process 0, Element 3', 'Process 0, Element 4', 
'Process 4, Element 0', 'Process 4, Element 1', 'Process 4, Element 2', 
'Process 4, Element 3', 'Process 4, Element 4', 'Process 3, Element 0', 
'Process 3, Element 1', 'Process 3, Element 2', 'Process 3, Element 3', 
'Process 3, Element 4']

Answered By: sebtac

Appending to the same list from different processes using multiprocessing

Question:

Answers: