How do I download a folder from AWS S3 with CloudPathLib from a public bucket?

Question:

I’m trying to download a folder from a public AWS S3 bucket using the Python library cloudpathlib. My code looks like this:

from cloudpathlib import CloudPath
path = r"C:somepathtofolder"
url = "s3://some-example-bucket/folder/"

cloud_path = CloudPath(url)
cloud_path.download_to(path)

Really straight forward. To my knowledge, this should work, because the bucket is public:

enter image description here

Here is the bucket policy (nabbed from the AWS S3 tutorial):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicRead",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion"
            ],
            "Resource": "arn:aws:s3:::cozy-auto-texture-sd-repo/*"
        }
    ]
}

When I run the Python code to install the folder stable-diffusion-v1-4 the following error message appears:

  File "E:Cozy-Auto-Texture-Filesvenvlibsite-packagesbotocoreauth.py", line 418, in add_auth
    raise NoCredentialsError()
botocore.exceptions.NoCredentialsError: Unable to locate credentials

My question is why is this happening? My bucket shouldn’t require credentials since it’s public. Is there something I’m missing with the buckets permissions or is it a Python code thing?

Thank you for reading and I appreciate the help!

Edit:

I’ve tried again with this method:

BUCKET_NAME = 'cozy-auto-texture-sd-repo'  # replace with your bucket name
KEY = 'stable-diffusion-v1-4'

s3 = boto3.resource('s3')

try:
    s3.Bucket(BUCKET_NAME).download_file(KEY, sd_path)
except botocore.exceptions.ClientError as e:
    if e.response['Error']['Code'] == "404":
        print("The object does not exist.")
    else:
        raise

However the same error message botocore.exceptions.NoCredentialsError: Unable to locate credentials appears, leading me to believe there is something wrong with my S3 bucket setup.

Asked By: Torrin worx

||
Source

Answers:

Assuming you are running this script locally, you may need to first set up your AWS credentials on your local machine.

See this existing answer:

It worked with an ~/.aws/credentials file containing:

[default]
aws_access_key_id=XXXXXXXXXXXXXX
aws_secret_access_key=YYYYYYYYYYYYYYYYYYYYYYYYYYY

Using aws configure also works if you have aws-cli installed.

Answered By: t_krill

It turns out that AWS S3 doesn’t allow you to download a folder from a public bucket. I got around this by simply zipping the folder I was trying to distribute and unzipping it in my application when the download was complete.

Answered By: Torrin worx

if you have aws cli you could do something using subprocess.call.

from pathlib import Path
path = Path(r"C:somepathtofolder")
import subprocess
url = CloudPath("s3://some-example-bucket/folder/")
subprocess.call(f"aws s3 sync {url.as_uri()} {path}", shell=True)
Note depending on what your preferences you may want call the command in a different way (subprocess.check_output() or shell=False, but command is split by ' ')
Answered By: Anton