Uploading large files with Python/Django
Question:
I am wondering if there are any ramifications in uploading files that are roughly 4GB in size through a web app using Django/Python? I remember in the past streaming uploads using Java was the preferred method but does this still today or is it perfectly safe to do so with Django/Python?
Answers:
Django will by default, put uploaded file data into memory if it is less than 2.5MB. Anything larger will be written to the server’s /tmp directory and then copied across when the transfer completes. Many of Django’s file upload settings can be customised, details are available in the documentation. You can also customise the file handling and you’ll certainly want to do this.
Before we consider any technical constraints, uploading such large files with the browser will give the user a very poor experience. There is no feedback about how the transfer is going (although google chrome does display the upload status as a percentage) and no way to pause or resume transfers.
You are also likely to run into problems on the server. Apart from the extremely long time that each thread will be taken with dealing with the streamed data, you have the time it takes for the system to copy the resulting file from /tmp to its correct location.
Unless you are very confident that you can foresee any problem that the server might have with the uploads, I would suggest that this is a bad idea. It’s pretty hard to find any information on this via google and there do seem to be a lot of hits that describe problems with large file uploads.
While Django is technically capable of receiving uploaded files this large, the very poor user experience and technical difficulties mean this may not be the best approach. Have you considered using dedicated software to handle the file transfer?
The last answer covers it. We routinely upload 2.5mb+ (but usually not 4gb)
adamnish link is correct, see this snippet (from his link to django docs) regarding writing the file to disk, instead of having it in memory first:
def handle_uploaded_file(f):
with open('some/file/name.txt', 'wb+') as destination:
for chunk in f.chunks():
destination.write(chunk)
More info on the “chunks” call: https://docs.djangoproject.com/en/dev/ref/files/uploads/#django.core.files.uploadedfile.UploadedFile.chunks
Page includes how to set “chunk” size, etc.
For future readers:
To up the max filesize allowed with in memory storage set the following in your settings.py:
FILE_UPLOAD_MAX_MEMORY_SIZE = 5242880 # make it 5Mb instead of 2Mb
Of course this won’t help you for 4Gb.
I am wondering if there are any ramifications in uploading files that are roughly 4GB in size through a web app using Django/Python? I remember in the past streaming uploads using Java was the preferred method but does this still today or is it perfectly safe to do so with Django/Python?
Django will by default, put uploaded file data into memory if it is less than 2.5MB. Anything larger will be written to the server’s /tmp directory and then copied across when the transfer completes. Many of Django’s file upload settings can be customised, details are available in the documentation. You can also customise the file handling and you’ll certainly want to do this.
Before we consider any technical constraints, uploading such large files with the browser will give the user a very poor experience. There is no feedback about how the transfer is going (although google chrome does display the upload status as a percentage) and no way to pause or resume transfers.
You are also likely to run into problems on the server. Apart from the extremely long time that each thread will be taken with dealing with the streamed data, you have the time it takes for the system to copy the resulting file from /tmp to its correct location.
Unless you are very confident that you can foresee any problem that the server might have with the uploads, I would suggest that this is a bad idea. It’s pretty hard to find any information on this via google and there do seem to be a lot of hits that describe problems with large file uploads.
While Django is technically capable of receiving uploaded files this large, the very poor user experience and technical difficulties mean this may not be the best approach. Have you considered using dedicated software to handle the file transfer?
The last answer covers it. We routinely upload 2.5mb+ (but usually not 4gb)
adamnish link is correct, see this snippet (from his link to django docs) regarding writing the file to disk, instead of having it in memory first:
def handle_uploaded_file(f):
with open('some/file/name.txt', 'wb+') as destination:
for chunk in f.chunks():
destination.write(chunk)
More info on the “chunks” call: https://docs.djangoproject.com/en/dev/ref/files/uploads/#django.core.files.uploadedfile.UploadedFile.chunks
Page includes how to set “chunk” size, etc.
For future readers:
To up the max filesize allowed with in memory storage set the following in your settings.py:
FILE_UPLOAD_MAX_MEMORY_SIZE = 5242880 # make it 5Mb instead of 2Mb
Of course this won’t help you for 4Gb.