Release redis lock if process die

Question

I’m using redis distributed locks to make sure some celery tasks don’t run concurrently. For managing locks with redis I’m using the python redis library version 4.0.0 (which specifies that any deadlock problems should be handled by the programmer).

Since I would like to avoid using the timeout parameter for the lock, since the task execution time is very variable, I would like to know if there is a way to verify that the active lock is actually owned by a process, and that it has not been blocked due to a sudden crash of the application and its failure to release.

@shared_task(
    name='test_task',
    queue="test"
)
def test(value):
    lock_acquired = False
    try:
        lock = redis_cli.lock('test_lock', sleep=1)
        lock_acquired = lock.acquire(blocking=False)        
        if lock_acquired:
            print(f"HELLO FROM: {value}")
            sleep(15)
        else:
            print(f"LOCKED")
    except Exception as e:
        print(str(e)) 
    finally:
        if lock_acquired:
            lock.release()

Taking the piece of code above, if the application were to unexpectedly crash during the sleep(15), the lock would still be locked at the next execution, even though the process that acquired it no longer exists. How can I prevent this situation?

python version: 3.6.8

redis server version: Redis server v=4.0.9 sha=00000000:0 malloc=jemalloc-3.6.0 bits=64 build=9435c3c2879311f3

celery==5.1.1

Asked By: Matteo Pasini

||

Source

Answer 1

Since I would like to avoid using the timeout parameter for the lock

In order to avoid holding the lock even if the client has crashed, you have to set a timeout.

since the task execution time is very variable, I would like to know if there is a way to verify that the active lock is actually owned by a process

You can use a watchdog thread to watch the lock. Say, you set a timeout of 5 seconds, and the watchdog thread extends the lock, i.e. renew the TTL, every 3 seconds. When your task is done, stop the watchdog thread, and delete the key, i.e. release the lock.

If your client crashes, the watchdog thread crashes too, and no longer extending the lock. When the key expires, others can hold the lock again.

I’m not familiar with Python client, but I write a C++ client which implements the above watchdog algorithm. You can take the doc for reference.

Answered By: for_stack

Release redis lock if process die

Question:

Answers: