Boto3 Copy_Object failing on size > 5GB

Question

I have the following function

async def _s3_copy_object(self, s3_source, s3_destination):
        source_bucket, source_key = get_s3_bucket_and_key(s3_source)
        destination_bucket, destination_key = get_s3_bucket_and_key(s3_destination)
    
        print("copying object: {} to {}".format(s3_source, s3_destination))
        source = {'Bucket': source_bucket, 'Key': source_key}
        await self._async_s3.copy_object(CopySource=source,
                                         Bucket=destination_bucket, Key=destination_key,
                                         ServerSideEncryption='AES256',
                                         MetadataDirective='COPY',
                                         TaggingDirective='COPY')

This works great if the file is under 5gb, but fails if the object is over 5gb.

I get the following error:

An error occurred (InvalidRequest) when calling the CopyObject operation: The specified copy source is larger than the maximum allowable size for a copy source: 5368709120: 1313

Is there a work around for this?

Asked By: user8128927

||

Source

Answer 1

You should consider using Multipart uploads.In a single operation, maximum allowed size is 5GB

Multi-part upload

Answered By: Pavan Kumar Kattamuri

Answer 2

You need to use the boto3 copy method instead of copy_object. It will perform the multi-part upload that is required when copying objects greater than 5GB. It will also handle the threading for you.

Answered By: dmulter

Boto3 Copy_Object failing on size > 5GB

Question:

Answers: