python queue & multiprocessing queue: how they behave?
Question:
This sample code works (I can write something in the file):
from multiprocessing import Process, Queue
queue = Queue()
def _printer(self, queue):
queue.put("hello world!!")
def _cmdDisp(self, queue):
f = file("Cmd.log", "w")
print >> f, queue.get()
f.close()
instead this other sample not: (errormsg: ‘module’ object is not callable)
import Queue
queue = Queue()
def _printer(self, queue):
queue.put("hello world!!")
def _cmdDisp(self, queue):
f = file("Cmd.log", "w")
print >> f, queue.get()
f.close()
this other sample not (I cannot write something in the file):
import Queue
queue = Queue.Queue()
def _printer(self, queue):
queue.put("hello world!!")
def _cmdDisp(self, queue):
f = file("Cmd.log", "w")
print >> f, queue.get()
f.close()
Can someone explain the differences? and the right to do?
Answers:
For your second example, you already gave the explanation yourself—Queue
is a module, which cannot be called.
For the third example: I assume that you use Queue.Queue
together with multiprocessing
. A Queue.Queue
will not be shared between processes. If the Queue.Queue
is declared before the processes then each process will receive a copy of it which is then independent of every other process. Items placed in the Queue.Queue
by the parent before starting the children will be available to each child. Items placed in the Queue.Queue
by the parent after starting the child will only be available to the parent. Queue.Queue
is made for data interchange between different threads inside the same process (using the threading module). The multiprocessing queues are for data interchange between different Python processes. While the API looks similar (it’s designed to be that way), the underlying mechanisms are fundamentally different.
multiprocessing
queues exchange data by pickling (serializing) objects and sending them through pipes.
Queue.Queue
uses a data structure that is shared between threads and locks/mutexes for correct behaviour.
Queue.Queue
-
Was created to work in concurrent environments spawned with the threading
module.
-
Each thread shares a reference to the Queue.Queue
object among them. No copying or serialization of data happens here and all the threads have access to the same data inside the queue.
multiprocessing.Queue
-
Was created to work in parallel environments spawned with the multiprocessing
module.
-
Each process gets access to a copy of the multiprocessing.Queue
object among them. The contents of the queue are copied across the processes via pickle serialization.
.
This sample code works (I can write something in the file):
from multiprocessing import Process, Queue
queue = Queue()
def _printer(self, queue):
queue.put("hello world!!")
def _cmdDisp(self, queue):
f = file("Cmd.log", "w")
print >> f, queue.get()
f.close()
instead this other sample not: (errormsg: ‘module’ object is not callable)
import Queue
queue = Queue()
def _printer(self, queue):
queue.put("hello world!!")
def _cmdDisp(self, queue):
f = file("Cmd.log", "w")
print >> f, queue.get()
f.close()
this other sample not (I cannot write something in the file):
import Queue
queue = Queue.Queue()
def _printer(self, queue):
queue.put("hello world!!")
def _cmdDisp(self, queue):
f = file("Cmd.log", "w")
print >> f, queue.get()
f.close()
Can someone explain the differences? and the right to do?
For your second example, you already gave the explanation yourself—Queue
is a module, which cannot be called.
For the third example: I assume that you use Queue.Queue
together with multiprocessing
. A Queue.Queue
will not be shared between processes. If the Queue.Queue
is declared before the processes then each process will receive a copy of it which is then independent of every other process. Items placed in the Queue.Queue
by the parent before starting the children will be available to each child. Items placed in the Queue.Queue
by the parent after starting the child will only be available to the parent. Queue.Queue
is made for data interchange between different threads inside the same process (using the threading module). The multiprocessing queues are for data interchange between different Python processes. While the API looks similar (it’s designed to be that way), the underlying mechanisms are fundamentally different.
multiprocessing
queues exchange data by pickling (serializing) objects and sending them through pipes.Queue.Queue
uses a data structure that is shared between threads and locks/mutexes for correct behaviour.
Queue.Queue
-
Was created to work in concurrent environments spawned with the
threading
module. -
Each thread shares a reference to the
Queue.Queue
object among them. No copying or serialization of data happens here and all the threads have access to the same data inside the queue.
multiprocessing.Queue
-
Was created to work in parallel environments spawned with the
multiprocessing
module. -
Each process gets access to a copy of the
multiprocessing.Queue
object among them. The contents of the queue are copied across the processes via pickle serialization.
.