How to use Celery to upload files in Django

Question

I was wondering how I can use Celery workers to handle file uploads.
So I tried implementing it on a simple class.
I overrided the create class in my ModelViewSet.
But apparently Django’s default json encoder does not serialize ImageFields (Lame).
I’ll really appreciate it if you guys could tell me how I can fix this.
Here is what I came up with:

serializers.py:

class ProductImageSerializer(serializers.ModelSerializer):
    class Meta:
        model = ProductImage
        fields = ['id', 'image']

tasks.py:

from time import sleep
from celery import shared_task
from .models import ProductImage

@shared_task:
def upload_image(product_id, image):
    print('Uploading image...')
    sleep(10)
    product = ProductImage(product_id=product_id, image=image)
    product.save()

views.py:

class ProductImageViewSet(ModelViewSet):
    serializer_class = ProductImageSerializer

    def get_queryset(self):
        return ProductImage.objects.filter(product_id=self.kwargs['product_pk'])

    def create(self, request, *args, **kwargs):
        product_id = self.kwargs['product_pk']
        image = self.request.FILES['image']
        image.open()
        image_data = Image.open(image)
        upload_image.delay(product_id, image_data)

        return Response('Thanks')

and here’s the my model containing my ImageField:

class ProductImage(models.Model):
    product = models.ForeignKey(Product, on_delete=models.CASCADE, related_name='images')
    image = models.ImageField(upload_to='store/images', validators=[validate_image_size])

Asked By: BFX0.9

||

Source

Answer 1

So I figured out a way to do this.
Here’s my solution:

The problem is that celery’s default json encoder cannot serialize Images, InMemoryUploadedFile, ModelObjects and…
So we need to pass it a value that is json serializable.
In this case, we wanna serialize an Image.
So what we can do is to convert our Image to bytes, then convert that bytes object to string, so we can send it to our celery task.
After we received the string in our task, we can convert it back to an Image and upload it using celery.Many people on the internet suggested this solution but none of them provided any code.
So here is the code for the example above, if you want to see it in action:

In my views.py I used a ModelViewSet and overrided the create method:

def create(self, request, *args, **kwargs):

        image = self.request.FILES['image'].read()

        byte = base64.b64encode(image)
        
        data = {
            'product_id': self.kwargs['product_pk'],
            'image': byte.decode('utf-8'),
            "name": self.request.FILES['image'].name
        }

        upload_image.delay(data=data)

        return Response('Uploading...')

And here’s my tasks.py:

from time import sleep
from celery import shared_task
from .models import ProductImage
import PIL.Image as Image
import io
import base64
import os
from django.core.files import File

@shared_task
def upload_image(data):
    
    print('Uploading image...')
    
    sleep(10)
    
    product_id = data['product_id']

    byte_data = data['image'].encode(encoding='utf-8')
    b = base64.b64decode(byte_data)
    img = Image.open(io.BytesIO(b))
    img.save(data['name'], format=img.format)
    
    with open(data['name'], 'rb') as file:
        picture = File(file)

        instance = ProductImage(product_id=product_id, image=picture)
        instance.save()
    
    os.remove(data['name'])

    print('Uploaded!')

I hope someone finds this helpful.
And anybody has any suggestions please let me know in the comments.
Have a nice day;)

Answered By: BFX0.9

Answer 2

Hello everyone earlier I posted a solution for this question and even though that solution worked properly, I found a better solution.
Encoding and Decoding binary files using base64 makes them larger and that is not something we want. So a better solution is to temporarily save the uploaded file on the disk, pass the path to our celery worker to upload it and create a ProductImage instance in our database and then delete the file we saved on the disk .

Here’s how to implement it:

tasks.py:

from time import sleep
from celery import shared_task
from .models import ProductImage
from django.core.files import File
from django.core.files.storage import FileSystemStorage
from pathlib import Path

@shared_task
def upload(product_id, path, file_name):

    print('Uploading image...')

    sleep(10)
    
    storage = FileSystemStorage()

    path_object = Path(path)

    with path_object.open(mode='rb') as file:
        
        picture = File(file, name=path_object.name)

        instance = ProductImage(product_id=product_id, image=picture)

        instance.save()


    storage.delete(file_name)

    print('Uploaded!')

In serializers.py you should override the create method of the ProductImage serializer like this:

    def create(self, validated_data):
        product_id = self.context['product_id']
        image_file = self.context['image_file']
        storage = FileSystemStorage()
        
        storage.save(image_file.name, File(image_file))

        return upload.delay(product_id=product_id, path=storage.path(image_file.name), file_name=image_file.name)

You should also override the create method in ProductImage’s ViewSet to provide the image file for your serializer’s context:

    def create(self, request, *args, **kwargs):
        product_id = self.kwargs['product_pk']
        image_file = self.request.FILES['image']
        serializer = ProductImageSerializer(
            data=request.data,
            context={
                'product_id': product_id,
                'image_file': image_file
            }
        )
        serializer.is_valid(raise_exception=True)
        serializer.save()
        return Response('Upload Started...')

Answered By: BFX0.9

How to use Celery to upload files in Django

Question:

Answers: