Copy and Rename files in S3 bucket Lambda
Question:
I am writing a lambda function to test the tables in Athena and I have gotten that part to work. I am trying to rename the ResultConfiguration outputlocation file name that the function creates which cannot be renamed unless it’s copied and deleted. I am getting Invalid Bucket Name everytime it tries to copy and I’m not sure why. This is the python code I have after doing some research:
def copy_delete_output_results(query_id, table_name):
oldCsvFileLocation = output + query_id + '.csv'
newOutputLocation = output + table_name
newFileName = table_name + '.csv'
s3.Object(newOutputLocation,newFileName).copy_from(CopySource=oldCsvFileLocation)
s3.Object(output,oldCsvFileLocation).delete()
oldCsvFileLocation = s3://ab-binaries-bucket-devtest/test/testQueryResults/1d8ab6ce-eda0-435f-93c3-2baa6fec8abd.csv
newOutputLocation = s3://ab-binaries-bucket-devtest/test/testQueryResults/dsm_abc_dns
newFileName = dsm_abc_dns.csv
Error Message:
Invalid bucket name "s3://ab-binaries-bucket-devtest/test/testQueryResults/dsm_abc_dns": Bucket name must match the regex "^[a-zA-Z0-9.-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z-0-9]*:[0-9]{12}:accesspoint[/:][a-zA-Z0-9-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9-]{1,63}$"
Answers:
s3.Object has two parameteres, the first one is the bucket name (string), and the second one is the file key (string).
For example:
import boto3
s3 = boto3.resource('s3')
source_bucket_name = 'ab-binaries-bucket-devtest'
source_file_key = 'test/testQueryResults/1d8ab6ce-eda0-435f-93c3-2baa6fec8abd.csv'
destination_bucket_name = 'ab-binaries-bucket-devtest'
destination_file_key = 'test/testQueryResults/dsm_abc_dns/dsm_abc_dns.csv'
# Copy the file
s3.Object(destination_bucket_name, destination_file_key).copy_from(
CopySource={'Bucket': source_bucket_name, 'Key': source_file_key}
)
# Delete the original file
s3.Object(source_bucket_name, source_file_key).delete()
So, the error you got is because you are passing the full s3 path in the first param to the s3.Object instead of passing the bucket-name
I am writing a lambda function to test the tables in Athena and I have gotten that part to work. I am trying to rename the ResultConfiguration outputlocation file name that the function creates which cannot be renamed unless it’s copied and deleted. I am getting Invalid Bucket Name everytime it tries to copy and I’m not sure why. This is the python code I have after doing some research:
def copy_delete_output_results(query_id, table_name):
oldCsvFileLocation = output + query_id + '.csv'
newOutputLocation = output + table_name
newFileName = table_name + '.csv'
s3.Object(newOutputLocation,newFileName).copy_from(CopySource=oldCsvFileLocation)
s3.Object(output,oldCsvFileLocation).delete()
oldCsvFileLocation = s3://ab-binaries-bucket-devtest/test/testQueryResults/1d8ab6ce-eda0-435f-93c3-2baa6fec8abd.csv
newOutputLocation = s3://ab-binaries-bucket-devtest/test/testQueryResults/dsm_abc_dns
newFileName = dsm_abc_dns.csv
Error Message:
Invalid bucket name "s3://ab-binaries-bucket-devtest/test/testQueryResults/dsm_abc_dns": Bucket name must match the regex "^[a-zA-Z0-9.-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z-0-9]*:[0-9]{12}:accesspoint[/:][a-zA-Z0-9-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9-]{1,63}$"
s3.Object has two parameteres, the first one is the bucket name (string), and the second one is the file key (string).
For example:
import boto3
s3 = boto3.resource('s3')
source_bucket_name = 'ab-binaries-bucket-devtest'
source_file_key = 'test/testQueryResults/1d8ab6ce-eda0-435f-93c3-2baa6fec8abd.csv'
destination_bucket_name = 'ab-binaries-bucket-devtest'
destination_file_key = 'test/testQueryResults/dsm_abc_dns/dsm_abc_dns.csv'
# Copy the file
s3.Object(destination_bucket_name, destination_file_key).copy_from(
CopySource={'Bucket': source_bucket_name, 'Key': source_file_key}
)
# Delete the original file
s3.Object(source_bucket_name, source_file_key).delete()
So, the error you got is because you are passing the full s3 path in the first param to the s3.Object instead of passing the bucket-name