Find out whether celery task exists

Question

Is it possible to find out whether a task with a certain task id exists? When I try to get the status, I will always get pending.

>>> AsyncResult('...').status
'PENDING'

I want to know whether a given task id is a real celery task id and not a random string. I want different results depending on whether there is a valid task for a certain id.

There may have been a valid task in the past with the same id but the results may have been deleted from the backend.

Asked By: dominik

||

Source

Answer 1

Try

AsyncResult('blubb').state

that may work.

It should return something different.

Answered By: Har

Answer 2

You need to call .get() on the AsyncTask object you create to actually fetch the result from the backend.

See the Celery FAQ.

To further clarify on my answer.

Any string is technically a valid ID, there is no way to validate the task ID. The only way to find out if a task exists is to ask the backend if it knows about it and to do that you must use .get().

This introduces the problem that .get() blocks when the backend doesn’t have any information about the task ID you supplied, this is by design to allow you to start a task and then wait for its completion.

In the case of the original question I’m going to assume that the OP wants to get the state of a previously completed task. To do that you can pass a very small timeout and catch timeout errors:

from celery.exceptions import TimeoutError
try:
    # fetch the result from the backend
    # your backend must be fast enough to return
    # results within 100ms (0.1 seconds)
    result = AsyncResult('blubb').get(timeout=0.1)
except TimeoutError:
    result = None

if result:
    print "Result exists; state=%s" % (result.state,)
else:
    print "Result does not exist"

It should go without saying that this only work if your backend is storing results, if it’s not there’s no way to know if a task ID is valid or not because nothing is keeping a record of them.

Even more clarification.

What you want to do cannot be accomplished using the AMQP backend because it does not store results, it forwards them.

My suggestion would be to switch to a database backend so that the results are in a database that you can query outside of the existing celery modules. If no tasks exist in the result database you can assume the ID is invalid.

Answered By: Evan Borgstrom

Answer 3

Please correct me if i’m wrong.

if built_in_status_check(task_id) == 'pending'
   if registry_exists(task_id) == true
      print 'Pending'
   else
      print 'Task does not exist'

Answered By: pravin

Answer 4

AsyncResult.state returns PENDING in case of unknown task ids.

PENDING

Task is waiting for execution or unknown. Any task id that is not
known is implied to be in the pending state.

http://docs.celeryproject.org/en/latest/userguide/tasks.html#pending

You can provide custom task ids if you need to distinguish unknown ids from existing ones:

>>> from tasks import add
>>> from celery.utils import uuid
>>> r = add.apply_async(args=[1, 2], task_id="celery-task-id-"+uuid())
>>> id = r.task_id
>>> id
'celery-task-id-b774c3f9-5280-4ebe-a770-14a6977090cd'
>>> if not "blubb".startswith("celery-task-id-"): print "Unknown task id"
... 
Unknown task id
>>> if not id.startswith("celery-task-id-"): print "Unknown task id"
...

Answered By: mher

Answer 5

Right now I’m using following scheme:

Get task id.
Set to memcache key like ‘task_%s’ % task.id message ‘Started’.
Pass task id to client.
Now from client I can monitor task status(set from task messages to memcache).
From task on ready – set to memcache key message ‘Ready’.
From client on task ready – start special task that will delete key from memcache and do necessary cleaning actions.

Answered By: Nikolay Fominyh

Answer 6

Celery does not write a state when the task is sent, this is partly an optimization (see the documentation).

If you really need it, it’s simple to add:

from celery import current_app
# `after_task_publish` is available in celery 3.1+
# for older versions use the deprecated `task_sent` signal
from celery.signals import after_task_publish

# when using celery versions older than 4.0, use body instead of headers

@after_task_publish.connect
def update_sent_state(sender=None, headers=None, **kwargs):
    # the task may not exist if sent using `send_task` which
    # sends tasks by name, so fall back to the default result backend
    # if that is the case.
    task = current_app.tasks.get(sender)
    backend = task.backend if task else current_app.backend
 
    backend.store_result(headers['id'], None, "SENT")

Then you can test for the PENDING state to detect that a task has not (seemingly)
been sent:

>>> result.state != "PENDING"

Answered By: asksol

Answer 7

So I have this idea:

import project.celery_tasks as tasks

def task_exist(task_id):
  found = False
  # tasks is my imported task module from celery
  # it is located under /project/project, where the settings.py file is located
  i = tasks.app.control.inspect()
  s = i.scheduled()
  for e in s:
    if task_id in s[e]:
      found = True
      break
  a = i.active()
  if not found:
    for e in a:
      if task_id in a[e]:
        found = True
        break
  r = i.reserved()
  if not found:
    for e in r:
      if task_id in r[e]:
        found = True
        break
  # if checking the status returns pending, yet we found it in any queues... it means it exists...
  # if it returns pending, yet we didn't find it on any of the queues... it doesn't exist
  return found

According to https://docs.celeryproject.org/en/stable/userguide/monitoring.html the different types of queue inspections are:
active,
scheduled,
reserved,
revoked,
registered,
stats,
query_task,

so pick and choose as you please.

And there might be a better way to go about checking the queues for their tasks, but this should work for me, for now.

Answered By: Shmack

Find out whether celery task exists

Question:

Answers: