how to put a function and arguments into python queue?

Question

I have a python program with 2 threads ( let’s name them ‘source’ and
‘destination’ ). Source thread sometimes post a message to destination
thread with some arguments. Than destination thread picks a message it
must call a corresponding function with aruments saved in message.

This task can be solved multiple ways. The easy one is tu put a big
‘if…if..if’ in destination thread’s message pick cycle and call
function according to received message type and saved arguments. But this
will result in huge amounf of code ( or big lookup table ) and adding new
messages / handler function will evolve additonal step to write code in
message pick cycle.

Since python treats functions as first-class objects and have tuples, i want
to put a function and argumens inside a message, so than destination thread
picks a message it just call a functon saved within a message without any
knowledge what function it is.

I can write a code for a functions with specified number of arguments:

from Queue import *
from thread import *
from time import *

q = Queue()

def HandleMsg( arg1, arg2 ) :
  print arg1, arg2

def HandleAnotherMsg( arg1, arg2, arg3 ) :
  print arg1, arg2, arg3

def DestinationThread( a ) :
  while True :
    (f, a, b) = q.get()
    f( a, b )

start_new_thread( DestinationThread, ( 0, ) )
print "start"
sleep( 1 )
q.put( (HandleMsg, 1, 2) )
sleep( 1 )
print "stop"

The question is: how to modify a code so i can put() a function with
any number of arguments in queue? for example HandleAnotherMsg() ?
Using q.put( (HandleAnotherMsg, 1, 2, 3) ) will rise a compilation error 🙁

Asked By: grigoryvp

||

Source

Answer 1

Why don’t you subclass Queue?


class MyQueue(Queue):
  # by using *args, you can have a variable number of arguments
  def put(self,*args):
    for arg in args:
       Queue.put(self,arg)

or, why don’t you put a list?


list = [function_obj]
for arg in function_args:
   list.append(arg)
queue.put(list)

Answered By: Geo

Answer 2

So simple:

def DestinationThread( a ) :
  while True :
    items = q.get()
    func = items[0]
    args = items[1:]
    func(*args)

Answered By: theller

Answer 3

from Queue import *
from thread import *
from time import *

q = Queue()

def HandleMsg( arg1, arg2 ) :
  print arg1, arg2

def HandleAnotherMsg( arg1, arg2, arg3 ) :
  print arg1, arg2, arg3

def DestinationThread() :
  while True :
    f, args = q.get()
    f(*args)

start_new_thread( DestinationThread, tuple() )
print "start"
sleep( 1 )
q.put( (HandleMsg, [1, 2]) )
sleep( 1 )
q.put( (HandleAnotherMsg, [1, 2, 3]) )
sleep( 1 )
print "stop"

Answered By: Toni Ruža

Answer 4

It sounds like you want to use the apply() intrinsic or its successor:

def f(x. y):
   print x+y

args = ( 1, 2 )

apply(f, args)   # old way

f(*args)        # new way

Answered By: Andy V

Answer 5

I’ve used a similar construct before:

class Call:
    def __init__(self, fn, *args, **kwargs):
        self.fn = fn
        self.args = args
        self.kwargs = kwargs

    def __call__(self):
        return self.fn(*self.args, **self.kwargs)


x = Call(zip, [0,1], [2,3], [4,5])

You should then be able to pass x to your other thread and call it from there:

x() # returns the same as zip([0,1], [2,3], [4,5])

Answered By: Jorenko

Answer 6

You can create an abstract message class with a run method. Then for each function that need to be transmitted via the queue, subclass and implement the function as the run method.
The sending thread will create an instance of the proper sub class and put it into the queue. The receiving thread will get an object from the queue and blindly execute the run method.

This is usually called the Command pattern (Gamma et al.)

Example:

class Message (object):
    """abstract message class"""
    def __init__(self, **kwargs):
        self.kwargs = kwargs

    def run(self):
        pass


class MessageOne (Message):
    """one message class"""
    def run(self):
         # perform this emssage's action using the kwargs

The sender will instantiate and send a message:

queue.put(MessageOne(one='Eins', two='Deux'))

The receiver simply gets a message object and execute it run method (without having to it..else.. thru the available types of messages):

msg = queue.get()
msg.run()

Answered By: Ber

Answer 7

Another interesting option is simply to pass in a lambda.

q.put(lambda: HandleMsg(1,2))
q.put(lambda: HandleAnother(8, "hello", extra="foo"))

def DestinationThread() :
   while True :
      f = q.get()
      f()

Answered By: cthulahoops

how to put a function and arguments into python queue?

Question:

Answers: