Global variable with Django and Celery

Question

I have a code like this,

wl_data = {}

def set_wl_data():
    global wl_data
    wl_data = get_watchlist_data()


def get_wl_data(scripcodes):
    # Filtering Data
    result = {scripcode:detail for scripcode, detail in wl_data.iteritems() if int(scripcode) in scripcodes or scripcode in scripcodes}
    return result

I am running this as a django project,
I am calling the setter method from celery, to update the global variable wl_data.
tastypie api will call the getter method get_wl_data to fetch global variable wl_data.

The problem is celery is updating wl_data properly.
But when we hit the tastypie api url in browser, the getter method
serves the old data.

There are so many related questions in stack overflow, but the difference here is setter method is called by celery task. Please help me to solve this issue.

Asked By: zeenfaiz

||

Source

Answer 1

If you’re doing anything with global variables in a Django project, you’re doing it wrong. In this case, Celery and Django are running in completely separate processes, so cannot share data. You need to get Celery to store that data somewhere – in the db, or a file – so that Django can pick it up and serve it.

Answered By: Daniel Roseman

Answer 2

The code below works with the global variable num for me in Django 3.1.7 and Celery 5.1.2. *However sometimes global variables don’t work properly between celery tasks which have different code from below so you should avoid to use global variables with celery tasks:

# "store/tasks.py"

from celery import shared_task

num = 0

@shared_task
def test1():
    global num
    num += 1
    return num

@shared_task
def test2():
    global num
    num += 1
    return num

@shared_task
def test3():
    global num
    num += 1
    return num

# "store/views.py"

from django.http import HttpResponse
from .tasks import test1, test2, test3

def test(request):
    test1.delay()
    test2.delay()
    test3.delay()
    return HttpResponse("Test")

Output:

Task store.tasks.test1[c222183b-73be-4fba-9813-be8141c6669c] received
Task store.tasks.test1[c222183b-73be-4fba-9813-be8141c6669c] succeeded in 0.0s: 1
Task store.tasks.test2[aa4bc9e5-95c6-4f8b-8122-3df273822ab5] received
Task store.tasks.test2[aa4bc9e5-95c6-4f8b-8122-3df273822ab5] succeeded in 0.0s: 2
Task store.tasks.test3[472727f3-368f-48ad-9d49-72f14962e8c5] received
Task store.tasks.test3[472727f3-368f-48ad-9d49-72f14962e8c5] succeeded in 0.0s: 3

Answered By: Kai – Kazuya Ito

Global variable with Django and Celery

Question:

Answers: