Upload Images to Amazon S3 using Django

Question

I’m currently resizing images on the fly when a user uploads a picture. The original picture is stored on Amazon S3 in a bucket called djangobucket. Inside this bucket, contains thousands of folders.

Each folder is named after the user. I don’t have to worry about bucket creation or folder creation since all of that is handled from the client side.

Here is a diagram:

djangobucket  ------------> bob ---------> picture1.jpg
                                           picture2.jpg
                                           picture3.jpg
                                           picture4.jpg

As you can see, Bob has many pictures. Once a user uploads a picture to S3, I download it through Django via a URL, in this case it would be: http://s3.amazonaws.com/djangobucket/bob/picture1.jpg

I download the image and perform image processing and save the processed image on my django app server.

I would like to send this processed image back into bob’s folder so that it can be publicly reached at http://s3.amazonaws.com/djangobucket/bob/picture1_processed.jpg

The client already has access to the amazon key and secret key so that he or she can upload pictures to this bucket. All users on the service use the same keys. I too will be using the same keys.

I’ve heard of something called Boto, but it involves Bucket creation and I’m unsure of how to do just the uploading part. I’m only concerned about uploading the picture to the appropriate user folder.

I’ve been researching this for hours so I’ve turned to the experts here.

Here is my code, just so that you can get a better understanding of what I’m doing.

user = 'bob'

url = 'http://s3.amazonaws.com/djangobucket/bob/picture1.jpg'
filename = url.split('/')[-1].split('.')[0]

download_photo = urllib.urlretrieve(url, "/home/ubuntu/Desktop/Resized_Images/%s.jpg" % (filename))
downloaded_photo = Image.open("/home/ubuntu/Desktop/Resized_Images/%s.jpg" % (filename))

resized_photo = downloaded_photo.resize((300, 300), Image.ANTIALIAS)
new_filename = filename + "_processed"

resized_photo.save("/home/ubuntu/Desktop/Resized_Images/%s.jpg" % (new_filename))

I would like to send the resized photo saved in /home/ubuntu/Desktop/Resized_Images/
to Bob’s folder in the djangobucket on Amazon S3 and make it publicly visible.

Thanks for your help.

EDIT
I found this link: http://www.laurentluce.com/posts/upload-and-download-files-tofrom-amazon-s3-using-pythondjango/

Not quite how to use it for my case, but I think I’m on the right track.

Asked By: deadlock

||

Source

Answer 1

boto is the best way to do this.

You can get an existing bucket using:

get_bucket(bucket_name, validate=True, headers=None)

After installing boto, this code should do what you need to do

from boto.s3.connection import S3Connection
from boto.s3.key import Key

conn = S3Connection('<aws access key>', '<aws secret key>')
bucket = conn.get_bucket('<bucket name>')
k = Key(bucket)
k.key = 'file_path_on_s3' # for example, 'images/bob/resized_image1.png'
k.set_contents_from_file(resized_photo)

Here are some links to the Boto API and Boto S3 Doc

Answered By: epau

Answer 2

What you can try..

1. Make sure you have the right S3 Permissions and Policy in place. Access ‘Policy’ via permissions tab in AWS S3 console for your specific bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<your_bucket_name>/*"
        },
        {
            "Sid": "Statement2",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::<your_bucket_name>/images/"
        }
    ]
}

2. Install WhiteNoiseMiddleware & django-storages in your project environment.

pip install whitenoise
pip install django-storages

3. Add the following to MIDDLEWARE= in settings.py

'whitenoise.middleware.WhiteNoiseMiddleware',

4. Following additions in settings.py are required to handle URLs from S3 correctly. The handling is done by django middleware & django-storages automatically

STATICFILES_LOCATION = 'static'
MEDIAFILES_LOCATION = 'media'

AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % os.environ['BUCKET_NAME']
AWS_ACCESS_KEY_ID = os.environ['AWS_KEY']
AWS_SECRET_ACCESS_KEY = os.environ['AWS_ACC_KEY']
AWS_STORAGE_BUCKET_NAME = os.environ['BUCKET_NAME']
AWS_S3_FILE_OVERWRITE = False
AWS_DEFAULT_ACL = None
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3StaticStorage'

MEDIA_ROOT = os.path.join (BASE_DIR, 'static/images/')
STATIC_ROOT = os.path.join (BASE_DIR, 'static')

STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, STATICFILES_LOCATION)
MEDIA_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, MEDIAFILES_LOCATION)

5. For aiming uploads into a precise S3 Bucket folder. (Additional)

In setting.py set media root:

MEDIA_ROOT = os.path.join (BASE_DIR, 'static/images/')

In models.py use ImageFiled and add upload_to= takes in a folder name and creates it with the first upload!:

image_variable = models.ImageField(null=True, default="{default_filename)", upload_to='uploads/')

Reference: django-storages , whiteNoiseMiddelware, S3 Access Troubleshooting

Answered By: zora

Upload Images to Amazon S3 using Django

Question:

Answers: