How to disallow pickle serialization in celery
Question:
Celery defaults to using pickle as its serialization method for tasks. As noted in the FAQ, this represents a security hole. Celery allows you to configure how tasks get serialized using the CELERY_TASK_SERIALIZER configuration parameter.
But this doesn’t solve the security problem. Even if tasks are serialized with JSON or similar, the workers will still execute tasks inserted into the queue with pickle serialization — they just respond to the content-type parameter in the message. So anybody who can write to the task queue can effectively pown the worker processes by writing malicious pickled objects.
How can I prevent the worker threads from running tasks serialized with pickle?
Answers:
I got an answer from the celery-users mailing list (From Ask Solem to be specific). Add these two lines to the config (celeryconfig/settings):
from kombu import serialization
serialization.registry._decoders.pop("application/x-python-serialize")
Now that Celery supports configuration on a per-app basis, there is a cleaner way to restrict the content that a consumer will execute.
c = celery.Celery()
c.conf.update(CELERY_ACCEPT_CONTENT = ['json'])
See the Celery docs on security for details, and for more advanced security options, such as signing content.
I was getting “ContentDisallowed: Refusing to deserialize untrusted content of type pickle (application/x-python-serialize)”
having:
CELERY_ACCEPT_CONTENT = ['json']
wasn’t enough… I had to also add the followings to settings:
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
Celery defaults to using pickle as its serialization method for tasks. As noted in the FAQ, this represents a security hole. Celery allows you to configure how tasks get serialized using the CELERY_TASK_SERIALIZER configuration parameter.
But this doesn’t solve the security problem. Even if tasks are serialized with JSON or similar, the workers will still execute tasks inserted into the queue with pickle serialization — they just respond to the content-type parameter in the message. So anybody who can write to the task queue can effectively pown the worker processes by writing malicious pickled objects.
How can I prevent the worker threads from running tasks serialized with pickle?
I got an answer from the celery-users mailing list (From Ask Solem to be specific). Add these two lines to the config (celeryconfig/settings):
from kombu import serialization
serialization.registry._decoders.pop("application/x-python-serialize")
Now that Celery supports configuration on a per-app basis, there is a cleaner way to restrict the content that a consumer will execute.
c = celery.Celery()
c.conf.update(CELERY_ACCEPT_CONTENT = ['json'])
See the Celery docs on security for details, and for more advanced security options, such as signing content.
I was getting “ContentDisallowed: Refusing to deserialize untrusted content of type pickle (application/x-python-serialize)”
having:
CELERY_ACCEPT_CONTENT = ['json']
wasn’t enough… I had to also add the followings to settings:
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'