Boto3 not uploading zip file to S3 python

Question

I’m trying to upload a .zip file to S3 using boto3 for python but the .zip file in my directory is not uploaded correctly.
The code downloads all emails of a given user, zips them in the same directory and uploads them to an S3 bucket.
The problem is that the file that gets uploaded is not the one I intend to upload. Instead a file with 18kb only appears.

Here’s the code:

import sys
import imaplib
import getpass
import email
import shutil
import boto3
import os

username = input("Enter user's first name: ")
surname = input("Enter user's surname: ")
email_address = username + "." + surname + "@gmail.com"
password = getpass.getpass()
directory = username + surname + '/'
def download_emails(server):
    result, data = server.uid('search', None, "ALL")    #search all email and return their uids
    if result == 'OK':
        for num in data[0].split():
            result, data = server.uid('fetch', num, '(RFC822)')    #RFC is a standard for the format of ARPA Internet text messages
            if result == 'OK':
                email_message = email.message_from_bytes(data[0][1])    #raw email text including headers
                file_name = email_message['Subject']       #use dates and file names(can be changed)
                if not os.path.exists(directory):
                    os.makedirs(directory)      #create a dir for user's emails
                try:
                    email_file = open(directory + file_name+'.eml', 'wb')   #open a file for each email and insert the data.
                    email_file.write(data[0][1])
                    email_file.close()
                except:
                    pass

#function to zip all the emails
def archive(zipname, directory):
    return shutil.make_archive(zipname, 'zip', root_dir=directory, base_dir=None)

#function to upload zipped emails to AWS bucket
def upload_to_s3(file_name):
    s3 = boto3.resource('s3',
                aws_access_key_id=accessKey,
                aws_secret_access_key=secretKey,
                aws_session_token=secretToken,
                )

    s3.Bucket('user-backups').put_object(Key=username.title() + " " +
                                surname.title() + "/" + file_name, Body=file_name)
    print("Uploaded")


def main():
    server = imaplib.IMAP4_SSL("imap.gmail.com", 993)   #connect to gmail's imap server
    server.login(email_address, password)   #enter creds
    result, data = server.select('"[Gmail]/All Mail"')  #get all emails(inbox, outbox etc)
    if result == 'OK':
        print("Downloading")
        download_emails(server)
        server.close()
    else:
        print("ERROR: Unable to open mailbox ", result)
    server.logout()
    archive(username + surname, directory)
    upload_to_s3(username + surname + ".zip")
    #os.remove(email_address + ".zip")
    #shutil.rmtree(email_address)
    print("Done")
if __name__ == "__main__":
    main()

Asked By: davidb

||

Source

Answer 1

The put_object function accepts Body which is either bytes object or a file object. You have currently just passed the plain filename (a string).

From documentation:

Body (bytes or seekable file-like object) — Object data.

So the fix should be to pass the file object. Consult this to know how to do that.

Answered By: Nabin

Answer 2

Just use s3.client.upload_file.

upload_file(Filename, Bucket, Key, ExtraArgs=None, Callback=None,
Config=None)

def upload_to_s3(file_name):
  s3 = boto3.client('s3')
  Key = username.title() + " " + surname.title() + "/" + file_name
  try: 
     s3.meta.client.upload_file('/path/to/file', 'user-backups', Key)
  except Exception as e:
     print(e)

Answered By: mootmoot

Answer 3

You can check out this article for more information.

There are a number of ways to upload. Check out this boto3 document where I have the methods listed below:

The managed upload methods are exposed in both the client and resource interfaces of boto3:

S3.Client method to upload a file by name: S3.Client.upload_file()
S3.Client method to upload a readable file-like object: S3.Client.upload_fileobj()
S3.Bucket method to upload a file by name: S3.Bucket.upload_file()
S3.Bucket method to upload a readable file-like object: S3.Bucket.upload_fileobj()
S3.Object method to upload a file by name: S3.Object.upload_file()
S3.Object method to upload a readable file-like object: S3.Object.upload_fileobj()

I made it work using s3.client.upload_file.

upload_file(Filename, Bucket, Key, ExtraArgs=None, Callback=None,
Config=None) .
Upload a file to an S3 object.

import boto3
s3Resource = boto3.resource('s3')

try: 
    s3Resource.meta.client.upload_file('/path/to/file', 'bucketName', 'keyName')
except Exception as err:
    print(err)

Answered By: Jun711

Answer 4

None of the above answers worked!
The following code worked for me..

 import os 
 def upload_file_zip(local_file_path):
   s3_client = boto3.client('s3')
   s3_path = os.path.join(os.path.basename(local_file_path))
   with open(local_file_path,mode='rb') as data:
      s3_client.upload_fileobj(data, BUCKET_NAME, s3_path)

Updated code as s3_folder parameter is not required here.

Answered By: Pradeep N.V.S

Answer 5

I managed to upload a .zip file by means of the following code:

def write_to_s3(filename, bucket, key):
    s3 = boto3.resource(service_name='s3',
                        aws_access_key_id=os.environ["AWS_USER1_ACCESS_KEY"],
                        aws_secret_access_key=os.environ["AWS_USER1_SECRET_ACCESS_KEY"])
    s3.meta.client.upload_file(filename, bucket, key)

Note : I had to use boto3.resource() instead of boto3.client() as was answered above by mootmoot, as it threw an Exception().

Exception thrown if boto3.client() is used:

AttributeError: 'ClientMeta' object has no attribute 'client'

Answered By: Timbus Calin

Boto3 not uploading zip file to S3 python

Question:

Answers: