Celery: How to ignore task result in chord or chain?

Question

I’m using celery, I have several tasks which needed to be executed in order.

For example I have this task:

@celery.task
def tprint(word):
    print word

And I want to do something like this:

>>> chain(tprint.s('a') | tprint.s('b'))()

Then I get TypeError: tprint() takes exactly 1 argument (2 given).

The same with chord, in this situation which I need a task to be executed after a group of tasks:

>>> chord([tprint.s('a'), tprint.s('b')])(tprint.s('c'))

So how to deal with this situation? I don’t care the result of each task, but they need to be executed in order.

Add a second parameter won’t work:

@celery.task
def tprint(word, ignore=None):
    print word

>>> chain(tprint.s('a', 0) | tprint.s('b'))()

This will print out ‘a’ and ‘None’.

Asked By: lxyu

||

Source

Answer 1

You can try doing something like this.
Instead of having a single parameter for function tprint you can have 2 parameters

def tprint(word, x=None):
    print word

then

chain(tprint.s('a', 0) | tprint.s('b'))()

Answered By: NitinJ

Answer 2

Finally find a workaround for this, a chain decorator will do this job.

I don’t know how exactly celery did it, but celery seems force bind previous task’s result to next task’s first argument.

So here’s an example:

def chain_deco(func):

    @functools.wraps(func)
    def wrapper(chain=None, *args, **kwargs):
        if chain is False:
            return False

        func(*args, **kwargs)
        return True

    return wrapper


@celery.task
@chain_deco
def hello(word):
    print "hello %s" % word

Now this will give the right output.

>>> (hello.s(word='a') | hello.s(word='b'))()

OR

>>> (hello.s('a') | hello.s('b'))(True)

And decorator also provide the ability to stop a chain in the middle(make the later cascade fail.)

The same mechanism should work for chord too.

Answered By: lxyu

Answer 3

There is a built-in functionality to ignore result in chaining and others – immutable subtask. You can use .si() shortcut instead of .s() or .subtask(immutable=True)

More details here: http://docs.celeryproject.org/en/master/userguide/canvas.html#immutability

Answered By: Vlad Frolov

Answer 4

One possible solution has already posted, but I’d like to add further clarification and an alternate solution (and in some cases a superior one).

The error you’re seeing, which indicates that your task’s signature needs to take into account a second parameter, is due to the fact that when calling tasks in a chain, Celery automatically pushes each tasks result as the first parameter of the following task.

From the docs:

Tasks can be linked together, which in practice means adding a callback task:

>>> res = add.apply_async((2, 2), link=mul.s(16))
>>> res.get()
4

The linked task will be applied with the result of its parent task as the first argument

Therefore, in your case, you could rewrite your task like this:

@celery.task
def tprint(result, word):
    print word

~~If you’re not going to do anything with the result, you may as well ignore it, by changing the decorator thus:~~

~~@celery.task(ignore_result=True)~~

~~And then you won’t have to change your task signature.~~

Sorry, that last point needs further research.

Answered By: ygesher

Celery: How to ignore task result in chord or chain?

Question:

Answers: