Alternative of threading.Timer?

Question

I have a producer-consumer pattern Queue, it consumes incoming events and schedule qualified events sending out in 5 seconds. I am using threading.Timer()python documentto do it and everything was working fine.

Recently, I was requested to change the scheduled time from 5 second to 30 minutes, and threading.Timer() crashes my script because previously the threads objects are created and are released very soon(only last 5 sec) but now it has to keep alive for 30 minutes.

Here’s the code:

 if scheduled_time and out_event:
     threading.Timer(scheduled_time, self.send_out_event, (socket_connection, received_event, out_event,)).start()  # schedule event send out

Can somesone shed some light on this? How can I solve this problem or is there any alternative for threading.Timer()?

Asked By: Haifeng Zhang

||

Source

Answer 1

Thanks for @dano ‘s comment about the 3rd party modules! Based on my work requirement, I didn’t install them on the server.

Instead of using threading.Timer(), I choose to use a Redis based Delay Queue, I found some helpful source online: A unique Python redis-based queue with delay. It solved my issue.

Briefly, the author creates a sorted set in redis and give it a name, add() would appends new data into the sorted set. Every time it pops at most one element from the sorted set based upon the epoch-time score, the element which holds qualified minimum score would be pop out (without removing from redis):

def add(self, received_event, delay_queue_name="delay_queue", delay=config.SECOND_RETRY_DELAY):
    try:
        score = int(time.time()) + delay
        self.__client.zadd(delay_queue_name, score, received_event)
        self.__logger.debug("added {0} to delay queue, delay time:{1}".format(received_event, delay))
    except Exception as e:
        self.__logger.error("error: {0}".format(e))


def pop(self, delay_queue_name="delay_queue"):
    min_score, max_score, element = 0, int(time.time()), None
    
    try:
        result = self.__client.zrangebyscore(delay_queue_name, min_score, max_score, start=0, num=1, withscores=False)
    except Exception as e:
        self.__logger.error("failed query from redis:{0}".format(e))
        return None

    if result and len(result) == 1:
        element = result[0]
        self.__logger.debug("poped {0} from delay queue".format(element))
    else:
        self.__logger.debug("no qualified element")
    return element

def remove(self, element, delay_queue_name="delay_queue"):
    self.__client.zrem(delay_queue_name, element)

self.__client is a Redis client instance, redis.StrictRedis(host=rhost,port=rport, db=rindex).

The difference between the online source with mine is that I switched zadd() parameters. The order of score and data are switched. Below is docs of zadd():

# SORTED SET COMMANDS
def zadd(self, name, *args, **kwargs):
    """
    Set any number of score, element-name pairs to the key ``name``. Pairs
    can be specified in two ways:

    As *args, in the form of: score1, name1, score2, name2, ...
    or as **kwargs, in the form of: name1=score1, name2=score2, ...

    The following example would add four values to the 'my-key' key:
    redis.zadd('my-key', 1.1, 'name1', 2.2, 'name2', name3=3.3, name4=4.4)
    """
    pieces = []
    if args:
        if len(args) % 2 != 0:
            raise RedisError("ZADD requires an equal number of "
                             "values and scores")
        pieces.extend(args)
    for pair in iteritems(kwargs):
        pieces.append(pair[1])
        pieces.append(pair[0])
    return self.execute_command('ZADD', name, *pieces)

Answered By: Haifeng Zhang

Alternative of threading.Timer?

Question:

Answers: