how do you configure `celery` to use serializer 'pickle'?

Question:

on the tasks.py side i do:

app = Celery(
        main='tasks',
        backend='rpc://',
        broker='pyamqp://USERNAME:PASSWORD@localhost',
        )
app.conf.task_serializer = 'pickle'
app.conf.result_serializer = 'pickle'
app.conf.event_serializer = 'pickle'
app.conf.accept_content = ['pickle']
app.conf.task_accept_content = ['pickle']
app.conf.result_accept_content = ['pickle']
app.conf.event_accept_content = ['pickle']

when i start two workers manually i via:

celery --app tasks worker --concurrency=1 -n by_hand_1 &
sleep 5
celery --app tasks worker --concurrency=1 -n by_hand_2

i get errors:

[ERROR/MainProcess] Refusing to deserialize disabled content of type pickle (application/x-python-serialize

how do you configure celery to use serializer ‘pickle’?

p.s. regarding "duplicates" the celery framework has changed significantly and since version 4.0 the default serializer was changed from pickle to default json… so that is why i am asking a new question.

Asked By: Trevor Boyd Smith

||
Source

Answers:

configuring the app with the MIME type seems to fix the issue:

app.conf.event_serializer = 'pickle' # this event_serializer is optional. somehow i missed this when writing this solution and it still worked without.
app.conf.task_serializer = 'pickle'
app.conf.result_serializer = 'pickle'
app.conf.accept_content = ['application/json', 'application/x-python-serialize']

previously i was trying to set the accept conect field with just ‘pickle’ which did not help.

p.s. here are some more notes regarding if you need the pickle serializer:

  • if you need to send json-non-serializable objects then you also need to configure the task serializer with ‘pickle’
  • if your results have json-non-serializable objects then you need to configure the result serializer to ‘pickle’
  • setting the accept_content to the value above means you can send both json and pickle and celery will see the type in the message headers and do the right thing
Answered By: Trevor Boyd Smith