Is there a way to move all files with specific file extensions from one folder to another in S3 using Boto3 python?

Question

I’m trying to move all files of specific file extensions from one folder to another using the S3 API.

The code I’m using to move files is:

s3_resource = boto3.resource(‘s3’)

s3_resource.Object(“bucket_name”, “newpath/to/object_B.txt”).copy_from(
 CopySource=”path/to/your/object_A.txt”)

s3_resource.Object(“bucket_name”, “path/to/your/object_A.txt”).delete()

Is there a way to modify this to move just the files that have specific extensions (e.g. .txt and .csv) ?

Asked By: Ricardo Francois

||

Source

Answer 1

It would be something like:

import boto3

s3_resource = boto3.resource('s3')

bucket = s3_resource.Bucket('bucket-name')

for object in bucket.objects.filter(Prefix='source_path/'):
  if object.key.endswith('.txt') or object.key.endswith('.csv'):
    bucket.Object(...).copy(...)
    object.delete(...)

You will need to modify the target key to include the destination path.

Answered By: John Rotenstein

Answer 2

Used some code from @John

import boto3
import json

def lambda_handler(event, context):
    s3 = boto3.resource('s3')
    src_bucket = s3.Bucket('mybucket-test-all-files')
    dest_bucket_jpg = s3.Bucket('mybucket-test-jpg-files')
    dest_bucket_png = s3.Bucket('mybucket-test-png-files')
    
    for object in src_bucket.objects.all():
        if object.key.endswith('.jpg') or object.key.endswith('.jpeg'):
            copy_source = {
                'Bucket': src_bucket,
                'Key': object.key
            }
            dest_bucket_jpg.copy(copy_source, object.key)
            object.delete()
        elif object.key.endswith('.png'):
            copy_source = {
                'Bucket': src_bucket,
                'Key': object.key
            }
            dest_bucket_png.copy(copy_source, object.key)
            object.delete()
        else: 
            print("Not copying any objects with diff file-types")
    # TODO implement
    return {
        'statusCode': 200,
        'body': json.dumps('Files Copied Successfully')
    }

Answered By: Kaustubh Mali

Is there a way to move all files with specific file extensions from one folder to another in S3 using Boto3 python?

Question:

Answers: