S3 appending random string in file name
Question:
I have a s3 folder with a csv file stored on it. I’m trying to download the last modified file. I’m using this script to get the last modified file:
s3_client = boto3.client('s3', aws_access_key_id=s3_extra_data['aws_access_key_id'],
aws_secret_access_key=s3_extra_data['aws_secret_access_key'])
response = s3_client.list_objects_v2(Bucket='test', Prefix='file_r/')
all = response['Contents']
latest = max(all, key=lambda x: x['LastModified'])
response = s3_client.list_objects_v2(Bucket='test', Prefix=latest["Key"])[:-52].lower())
all = response['Contents']
latest = max(all, key=lambda x: x['LastModified'])
print("LATEST ->" + str(latest["Key"])[:-52].lower())
print("PATH ->" + str(latest["Key"]))
s3_client.download_file("test", latest["Key"], str(latest["Key"]))
This code lists my last modified object, the file name is part-00000-40f267f2-38dc-4bab-811c-4c3052fdb1ba-c000.csv and is inside the file_r folder.
Although, when I use s3_client.download_file i get the following error:
'file_r/part-00000-40f267f2-38dc-4bab-811c-4c3052fdb1ba-c000.csv.8cEebaeb'
When i print my path and my file I get the correct values
LATEST -> file_r/part
PATH -> file_r/part-00000-40f267f2-38dc-4bab-811c-4c3052fdb1ba-c000.csv
Why the value .8cEebaeb is appended after the .csv extension since the PATH is correct.
Any thoughs on that?
Answers:
To solve the problem I changed the code to:
s3_client = boto3.client('s3', aws_access_key_id=s3_extra_data['aws_access_key_id'],
aws_secret_access_key=s3_extra_data['aws_secret_access_key'])
response = s3_client.list_objects_v2(Bucket='test', Prefix='file_r/')
all = response['Contents']
latest = max(all, key=lambda x: x['LastModified'])
s3_client.download_file("test", latest["Key"], "FILE_NAME"))
I had this issue when the local folder is not created.
folder1/folder2/filename.foo
If folder1 or folder2 does not exist locally, boto3 returns error :
FileNotFoundError: [Errno 2] No such file or directory: 'folder1/folder2/finemane.foo.F89bdcAc'
Hi Please note that this issue can sometime happen due to number of char in the folder path in windows machines including the file name. I faced similar issue and when I tried to save to the same folder path it was giving me error till the time I reduced the number of char to 228.
I have a s3 folder with a csv file stored on it. I’m trying to download the last modified file. I’m using this script to get the last modified file:
s3_client = boto3.client('s3', aws_access_key_id=s3_extra_data['aws_access_key_id'],
aws_secret_access_key=s3_extra_data['aws_secret_access_key'])
response = s3_client.list_objects_v2(Bucket='test', Prefix='file_r/')
all = response['Contents']
latest = max(all, key=lambda x: x['LastModified'])
response = s3_client.list_objects_v2(Bucket='test', Prefix=latest["Key"])[:-52].lower())
all = response['Contents']
latest = max(all, key=lambda x: x['LastModified'])
print("LATEST ->" + str(latest["Key"])[:-52].lower())
print("PATH ->" + str(latest["Key"]))
s3_client.download_file("test", latest["Key"], str(latest["Key"]))
This code lists my last modified object, the file name is part-00000-40f267f2-38dc-4bab-811c-4c3052fdb1ba-c000.csv and is inside the file_r folder.
Although, when I use s3_client.download_file i get the following error:
'file_r/part-00000-40f267f2-38dc-4bab-811c-4c3052fdb1ba-c000.csv.8cEebaeb'
When i print my path and my file I get the correct values
LATEST -> file_r/part
PATH -> file_r/part-00000-40f267f2-38dc-4bab-811c-4c3052fdb1ba-c000.csv
Why the value .8cEebaeb is appended after the .csv extension since the PATH is correct.
Any thoughs on that?
To solve the problem I changed the code to:
s3_client = boto3.client('s3', aws_access_key_id=s3_extra_data['aws_access_key_id'],
aws_secret_access_key=s3_extra_data['aws_secret_access_key'])
response = s3_client.list_objects_v2(Bucket='test', Prefix='file_r/')
all = response['Contents']
latest = max(all, key=lambda x: x['LastModified'])
s3_client.download_file("test", latest["Key"], "FILE_NAME"))
I had this issue when the local folder is not created.
folder1/folder2/filename.foo
If folder1 or folder2 does not exist locally, boto3 returns error :
FileNotFoundError: [Errno 2] No such file or directory: 'folder1/folder2/finemane.foo.F89bdcAc'
Hi Please note that this issue can sometime happen due to number of char in the folder path in windows machines including the file name. I faced similar issue and when I tried to save to the same folder path it was giving me error till the time I reduced the number of char to 228.