Set Django's FileField to an existing file
Question:
I have an existing file on disk (say /folder/file.txt) and a FileField model field in Django.
When I do
instance.field = File(file('/folder/file.txt'))
instance.save()
it re-saves the file as file_1.txt (the next time it’s _2, etc.).
I understand why, but I don’t want this behavior – I know the file I want the field to be associated with is really there waiting for me, and I just want Django to point to it.
How?
Answers:
I had exactly the same problem! then I realize that my Models were causing that. example I hade my models like this:
class Tile(models.Model):
image = models.ImageField()
Then, I wanted to have more the one tile referencing the same file in the disk! The way that I found to solve that was change my Model structure to this:
class Tile(models.Model):
image = models.ForeignKey(TileImage)
class TileImage(models.Model):
image = models.ImageField()
Which after I realize that make more sense, because if I want the same file being saved more then one in my DB I have to create another table for it!
I guess you can solve your problem like that too, just hoping that you can change the models!
EDIT
Also I guess you can use a different storage, like this for instance: SymlinkOrCopyStorage
http://code.welldev.org/django-storages/src/11bef0c2a410/storages/backends/symlinkorcopy.py
If you want to do this permanently, you need to create your own FileStorage class
import os
from django.conf import settings
from django.core.files.storage import FileSystemStorage
class MyFileStorage(FileSystemStorage):
# This method is actually defined in Storage
def get_available_name(self, name):
if self.exists(name):
os.remove(os.path.join(settings.MEDIA_ROOT, name))
return name # simply returns the name passed
Now in your model, you use your modified MyFileStorage
from mystuff.customs import MyFileStorage
mfs = MyFileStorage()
class SomeModel(model.Model):
my_file = model.FileField(storage=mfs)
It’s right to write own storage class. However get_available_name is not the right method to override.
get_available_name is called when Django sees a file with same name and tries to get a new available file name. It’s not the method that causes the rename. the method caused that is _save. Comments in _save is pretty good and you can easily find it opens file for writing with flag os.O_EXCL which will throw an OSError if same file name already exists. Django catches this Error then calls get_available_name to get a new name.
So I think the correct way is to override _save and call os.open() without flag os.O_EXCL. The modification is quite simple however the method is a little be long so I don’t paste it here. Tell me if you need more help 🙂
just set instance.field.name to the path of your file
e.g.
class Document(models.Model):
file = FileField(upload_to=get_document_path)
description = CharField(max_length=100)
doc = Document()
doc.file.name = 'path/to/file' # must be relative to MEDIA_ROOT
doc.file
<FieldFile: path/to/file>
try this (doc):
instance.field.name = <PATH RELATIVE TO MEDIA_ROOT>
instance.save()
You should define your own storage, inherit it from FileSystemStorage, and override OS_OPEN_FLAGS class attribute and get_available_name() method:
Django Version: 3.1
Project/core/files/storages/backends/local.py
import os
from django.core.files.storage import FileSystemStorage
class OverwriteStorage(FileSystemStorage):
"""
FileSystemStorage subclass that allows overwrite the already existing
files.
Be careful using this class, as user-uploaded files will overwrite
already existing files.
"""
# The combination that don't makes os.open() raise OSError if the
# file already exists before it's opened.
OS_OPEN_FLAGS = os.O_WRONLY | os.O_TRUNC | os.O_CREAT | getattr(os, 'O_BINARY', 0)
def get_available_name(self, name, max_length=None):
"""
This method will be called before starting the save process.
"""
return name
In your model, use your custom OverwriteStorage
myapp/models.py
from django.db import models
from core.files.storages.backends.local import OverwriteStorage
class MyModel(models.Model):
my_file = models.FileField(storage=OverwriteStorage())
The answers work fine if you are using the app’s filesystem to store your files. But, If your are using boto3 and uploading to sth like AWS S3 and maybe you want to set a file already existing in an S3 bucket to your model’s FileField then, this is what you need.
We have a simple model class with a filefield:
class Image(models.Model):
img = models.FileField()
owner = models.ForeignKey(get_user_model(), on_delete=models.CASCADE, related_name='images')
date_added = models.DateTimeField(editable=False)
date_modified = models.DateTimeField(editable=True)
from botocore.exceptions import ClientError
import boto3
s3 = boto3.client(
's3',
aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY")
)
s3_key = S3_DIR + '/' + filename
bucket_name = os.getenv("AWS_STORAGE_BUCKET_NAME")
try:
s3.upload_file(local_file_path, bucket_name, s3_key)
# we want to store it to our db model called **Image** after s3 upload is complete so,
image_data = Image()
image_data.img.name = s3_key # this does it !!
image_data.owner = get_user_model().objects.get(id=owner_id)
image_data.save()
except ClientError as e:
print(f"failed uploading to s3 {e}")
Setting the S3 KEY into the name field of the FileField does the trick. As much i have tested everything related works as expected e.g previewing the image file in django admin. fetching the images from db appends the root s3 bucket prefix (or, the cloudfront cdn prefix) to the s3 keys of the files too. Ofcourse, its given that, i already had a working setup of the django settings.py for boto and s3.
I have an existing file on disk (say /folder/file.txt) and a FileField model field in Django.
When I do
instance.field = File(file('/folder/file.txt'))
instance.save()
it re-saves the file as file_1.txt (the next time it’s _2, etc.).
I understand why, but I don’t want this behavior – I know the file I want the field to be associated with is really there waiting for me, and I just want Django to point to it.
How?
I had exactly the same problem! then I realize that my Models were causing that. example I hade my models like this:
class Tile(models.Model):
image = models.ImageField()
Then, I wanted to have more the one tile referencing the same file in the disk! The way that I found to solve that was change my Model structure to this:
class Tile(models.Model):
image = models.ForeignKey(TileImage)
class TileImage(models.Model):
image = models.ImageField()
Which after I realize that make more sense, because if I want the same file being saved more then one in my DB I have to create another table for it!
I guess you can solve your problem like that too, just hoping that you can change the models!
EDIT
Also I guess you can use a different storage, like this for instance: SymlinkOrCopyStorage
http://code.welldev.org/django-storages/src/11bef0c2a410/storages/backends/symlinkorcopy.py
If you want to do this permanently, you need to create your own FileStorage class
import os
from django.conf import settings
from django.core.files.storage import FileSystemStorage
class MyFileStorage(FileSystemStorage):
# This method is actually defined in Storage
def get_available_name(self, name):
if self.exists(name):
os.remove(os.path.join(settings.MEDIA_ROOT, name))
return name # simply returns the name passed
Now in your model, you use your modified MyFileStorage
from mystuff.customs import MyFileStorage
mfs = MyFileStorage()
class SomeModel(model.Model):
my_file = model.FileField(storage=mfs)
It’s right to write own storage class. However get_available_name is not the right method to override.
get_available_name is called when Django sees a file with same name and tries to get a new available file name. It’s not the method that causes the rename. the method caused that is _save. Comments in _save is pretty good and you can easily find it opens file for writing with flag os.O_EXCL which will throw an OSError if same file name already exists. Django catches this Error then calls get_available_name to get a new name.
So I think the correct way is to override _save and call os.open() without flag os.O_EXCL. The modification is quite simple however the method is a little be long so I don’t paste it here. Tell me if you need more help 🙂
just set instance.field.name to the path of your file
e.g.
class Document(models.Model):
file = FileField(upload_to=get_document_path)
description = CharField(max_length=100)
doc = Document()
doc.file.name = 'path/to/file' # must be relative to MEDIA_ROOT
doc.file
<FieldFile: path/to/file>
try this (doc):
instance.field.name = <PATH RELATIVE TO MEDIA_ROOT>
instance.save()
You should define your own storage, inherit it from FileSystemStorage, and override OS_OPEN_FLAGS class attribute and get_available_name() method:
Django Version: 3.1
Project/core/files/storages/backends/local.py
import os
from django.core.files.storage import FileSystemStorage
class OverwriteStorage(FileSystemStorage):
"""
FileSystemStorage subclass that allows overwrite the already existing
files.
Be careful using this class, as user-uploaded files will overwrite
already existing files.
"""
# The combination that don't makes os.open() raise OSError if the
# file already exists before it's opened.
OS_OPEN_FLAGS = os.O_WRONLY | os.O_TRUNC | os.O_CREAT | getattr(os, 'O_BINARY', 0)
def get_available_name(self, name, max_length=None):
"""
This method will be called before starting the save process.
"""
return name
In your model, use your custom OverwriteStorage
myapp/models.py
from django.db import models
from core.files.storages.backends.local import OverwriteStorage
class MyModel(models.Model):
my_file = models.FileField(storage=OverwriteStorage())
The answers work fine if you are using the app’s filesystem to store your files. But, If your are using boto3 and uploading to sth like AWS S3 and maybe you want to set a file already existing in an S3 bucket to your model’s FileField then, this is what you need.
We have a simple model class with a filefield:
class Image(models.Model):
img = models.FileField()
owner = models.ForeignKey(get_user_model(), on_delete=models.CASCADE, related_name='images')
date_added = models.DateTimeField(editable=False)
date_modified = models.DateTimeField(editable=True)
from botocore.exceptions import ClientError
import boto3
s3 = boto3.client(
's3',
aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY")
)
s3_key = S3_DIR + '/' + filename
bucket_name = os.getenv("AWS_STORAGE_BUCKET_NAME")
try:
s3.upload_file(local_file_path, bucket_name, s3_key)
# we want to store it to our db model called **Image** after s3 upload is complete so,
image_data = Image()
image_data.img.name = s3_key # this does it !!
image_data.owner = get_user_model().objects.get(id=owner_id)
image_data.save()
except ClientError as e:
print(f"failed uploading to s3 {e}")
Setting the S3 KEY into the name field of the FileField does the trick. As much i have tested everything related works as expected e.g previewing the image file in django admin. fetching the images from db appends the root s3 bucket prefix (or, the cloudfront cdn prefix) to the s3 keys of the files too. Ofcourse, its given that, i already had a working setup of the django settings.py for boto and s3.