Uploading Files to AWS S3 Bucket Folder in Python Causes Regex Error

Question

I have an AWS S3 bucket called task-details and a folder called archive so the S3 URI is s3://task-details/archive/ & ARN of arn:aws:s3:::task-details/archive/. I’m trying to use the upload_file method from Python’s boto3 package to upload a CSV file to a folder within the S3 bucket.

Below is the method I am using to try and upload data to the folder but I keep getting a regex error which makes me think that I can’t upload data to a specific folder in a bucket. Does anyone know how to do this?

Method:

import logging
import boto3
from botocore.exceptions import ClientError

def upload_file(file_name, bucket, object_name=None):
    """Upload a file to an S3 bucket

    :param file_name: File to upload
    :param bucket: Bucket to upload to
    :param object_name: S3 object name. If not specified then file_name is used
    :return: True if file was uploaded, else False
    """

    # If S3 object_name was not specified, use file_name
    if object_name is None:
        object_name = file_name

    # Upload the file
    s3_client = boto3.client('s3')
    try:
        response = s3_client.upload_file(file_name, bucket, object_name)
    except ClientError as e:
        logging.error(e)
        return False
    return True

My Code (I’ve also tried bucket = s3://task-details/archive/ and bucket = task-details/archive/):

upload_file(
    file_name = filepath, 
    bucket = "arn:aws:s3:::task-details/archive/", 
    object_name = object_name
)

The Error:

Invalid bucket name "arn:aws:s3:::task-details/archive/": Bucket name must match the regex 
"^[a-zA-Z0-9.-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z-0-9]+:[0-9]{12}:accesspoint[/:][a-zA-Z0-9-]{1,63}$|^arn:(aws).*:s3-outposts:[a-z-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9-]{1,63}$"

Asked By: Austin Ulfers

||

Source

Answer 1

The API call wants the bucket name or ARN. Your bucket name is task-details and your bucket ARN is arn:aws:s3:::task-details.

You use the Key parameter when calling upload_file to specify the object’s key, for example archive/cats/persian.png. Note that the S3 object key is not simply the object/file name but also includes the prefix/folder.

Answered By: jarmod

Answer 2

The problem was that I needed to add the folder path as part of the object_name instead of the bucket.

Fixed Code:

upload_file(
    file_name = filepath, 
    bucket = "arn:aws:s3:::task-details", 
    object_name = "archive/" + object_name
)

Answered By: Austin Ulfers

Answer 3

In my case, the other answers didn’t resolve the error. I had to change my bucket name from the arn ("arn:aws:s3:::bucket-name") to the bucket name alone ("bucket-name").

Adding a potentially helpful explanation from @jarmod:

Interesting that the current boto3 docs say it’s the bucket name but the error message in the original post above indicated that it could be an ARN. Personally I always use bucket name and maybe the error message above is a red herring. Certainly it seems that ARN is not supported.

Answered By: 208_man

Answer 4

I was trying to upload the contents of a zip file. Also I wanted to upload to a folder inside the bucket. So this is my working solution.

s3.upload_fileobj(
       Bucket='user-data',
       Key='my_extracted-files/' + file_name, # file_name is the name of file
       Fileobj=zop # This is a file obj
      )

My guess is the naming format of bucket and key params is the same for method: upload_file too

Note: Don’t forget to add Put permissions in your policy

Answered By: Arindam Roychowdhury

Uploading Files to AWS S3 Bucket Folder in Python Causes Regex Error

Question:

Answers: