Override S3 endpoint using Boto3 configuration file

Question:

OVERVIEW:

I’m trying to override certain variables in boto3 using the configuration file (~/aws/confg).
In my use case I want to use fakes3 service and send S3 requests to the localhost.

EXAMPLE:

In boto (not boto3), I can create a config in ~/.boto similar to this one:

[s3]
host = localhost
calling_format = boto.s3.connection.OrdinaryCallingFormat

[Boto]
is_secure = False

And the client can successfully pick up desired changes and instead of sending traffic to the real S3 service, it will send it to the localhost.

>>> import boto
>>> boto.connect_s3()
S3Connection:localhost
>>>

WHAT I TRIED:

I’m trying to achieve a similar result using boto3 library. By looking at the source code I found that I can use ~/aws/config location. I’ve also found an example config in unittests folder of botocore.

I tried to modify the config to achieve the desired behaviour. But unfortunately, it doesn’t work.

Here is the config:

[default]
aws_access_key_id = XXXXXXXXX
aws_secret_access_key = YYYYYYYYYYYYYY
region = us-east-1
is_secure = False
s3 =
    host = localhost

QUESTION:

  1. How to overwrite clients variables using config file?
  2. Where can I find a complete list of allowed variables for the configuration?
Asked By: Vor

||
Source

Answers:

boto3 only reads the signature version for s3 from that config file. You may want to open a feature request, but for now here is how you can address a custom endpoint:

import boto3
from botocore.utils import fix_s3_host
resource = boto3.resource(service_name='s3', endpoint_url='http://localhost')
resource.meta.client.meta.events.unregister('before-sign.s3', fix_s3_host)

That bit about the meta is important because boto3 automatically changes the endpoint to your_bucket_name.s3.amazonaws.com when it sees fit 1. If you’ll be working with both your own host and s3, you may wish to override the functionality rather than removing it altogether.

Answered By: Jordon Phillips

You cannot set host in config file, however you can override it from your code with boto3.

import boto3

session = boto3.session.Session()

s3_client = session.client(
    service_name='s3',
    aws_access_key_id='aaa',
    aws_secret_access_key='bbb',
    endpoint_url='http://localhost',
)

Then you can interact as usual.

print(s3_client.list_buckets())
Answered By: Chillar Anand

Another way:

import boto3

s3client = boto3.client('s3', endpoint_url='http://X.X.x.X:8080/',
        aws_access_key_id = 'XXXXXXX',
        aws_secret_access_key = 'XXXXXXXX')
bucket_name = 'aaaaa'
s3client.create_bucket(Bucket=bucket_name)
Answered By: petertc

using boto3 resource:

import boto3

# use third party object storage
s3 = boto3.resource('s3', endpoint_url='https://URL:443',
  aws_access_key_id = 'AccessKey',
  aws_secret_access_key = 'SecertKey')

# Print out bucket names
for bucket in s3.buckets.all():
 print(bucket.name)
Answered By: Andi-OTC

I’ve just submitted a PR to boto3 to add an env var to allow you to override the endpoint_url if you need to use a dependent module which uses boto3 (in which case you may not be able to modify the call to boto3.client directly)

https://github.com/boto/boto3/pull/2746

https://github.com/rwillmer/boto3

Answered By: Rachel
s3_client = boto3.client(
    "s3",
    "us-east-1",
    aws_access_key_id="test_id",
    aws_secret_access_key="test_key",
    endpoint_url="https://localhost:7000",
    use_ssl=True,
    verify=False,
)
Answered By: Q. Qiao