Difference in boto3 between resource, client, and session?
Question:
I am using Python 2.7.12 in Ubuntu 16.04 LTS. I’m learning how to use boto3 from the following link: https://boto3.readthedocs.io/en/latest/guide/quickstart.html#using-boto-3. My doubt is when to use resource, client, or session, and their respective functionality.
Answers:
I’ll try and explain it as simple as possible. So there is no guarantee of the accuracy of the actual terms.
Session is where to initiate the connectivity to AWS services. E.g. following is default session that uses the default credential profile(e.g. ~/.aws/credentials, or assume your EC2 using IAM instance profile )
sqs = boto3.client('sqs')
s3 = boto3.resource('s3')
Because default session is limit to the profile or instance profile used, sometimes you need to use the custom session to override the default session configuration (e.g. region_name, endpoint_url, etc. ) e.g.
# custom resource session must use boto3.Session to do the override
my_west_session = boto3.Session(region_name = 'us-west-2')
my_east_session = boto3.Session(region_name = 'us-east-1')
backup_s3 = my_west_session.resource('s3')
video_s3 = my_east_session.resource('s3')
# you have two choices of create custom client session.
backup_s3c = my_west_session.client('s3')
video_s3c = boto3.client("s3", region_name = 'us-east-1')
Resource : This is the high-level service class recommended to be used. This allows you to tied particular AWS resources and passes it along, so you just use this abstraction than worry which target services are pointed to. As you notice from the session part, if you have a custom session, you just pass this abstract object than worrying about all custom regions,etc to pass along. Following is a complicated example
E.g.
import boto3
my_west_session = boto3.Session(region_name = 'us-west-2')
my_east_session = boto3.Session(region_name = 'us-east-1')
backup_s3 = my_west_session.resource("s3")
video_s3 = my_east_session.resource("s3")
backup_bucket = backup_s3.Bucket('backupbucket')
video_bucket = video_s3.Bucket('videobucket')
# just pass the instantiated bucket object
def list_bucket_contents(bucket):
for object in bucket.objects.all():
print(object.key)
list_bucket_contents(backup_bucket)
list_bucket_contents(video_bucket)
Client is a low level class object. For each client call, you need to explicitly specify the targeting resources, the designated service target name must be pass long. You will lose the abstraction ability.
For example, if you only deal with the default session, this looks similar to boto3.resource.
import boto3
s3 = boto3.client('s3')
def list_bucket_contents(bucket_name):
for object in s3.list_objects_v2(Bucket=bucket_name) :
print(object.key)
list_bucket_contents('Mybucket')
However, if you want to list objects from a bucket in different regions, you need to specify the explicit bucket parameter required for the client.
import boto3
backup_s3 = my_west_session.client('s3',region_name = 'us-west-2')
video_s3 = my_east_session.client('s3',region_name = 'us-east-1')
# you must pass boto3.Session.client and the bucket name
def list_bucket_contents(s3session, bucket_name):
response = s3session.list_objects_v2(Bucket=bucket_name)
if 'Contents' in response:
for obj in response['Contents']:
print(obj['key'])
list_bucket_contents(backup_s3, 'backupbucket')
list_bucket_contents(video_s3 , 'videobucket')
Client and Resource are two different abstractions within the boto3 SDK for making AWS service requests. If you want to make API calls to an AWS service with boto3, then you do so via a Client or a Resource.
You would typically choose to use either the Client abstraction or the Resource abstraction, but you can use both, as needed. I’ve outlined the differences below to help readers decide which to use.
Session is largely orthogonal to the concepts of Client and Resource (but is used by both).
Here’s some more detailed information on what Client, Resource, and Session are all about.
Client:
- this is the original boto3 API abstraction
- it provides low-level AWS service access
- all AWS service operations are supported by clients
- it exposes botocore client to the developer
- it typically maps 1:1 with the AWS service API
- it exposes snake-cased method names (e.g. ListBuckets API => list_buckets method)
- typically yields primitive, non-marshalled data (e.g. DynamoDB attributes are dicts representing primitive DynamoDB values)
- requires you to code result pagination
- it is generated from an AWS service description
Here’s an example of client-level access to an S3 bucket’s objects:
import boto3
client = boto3.client('s3')
response = client.list_objects_v2(Bucket='mybucket')
for content in response['Contents']:
obj_dict = client.get_object(Bucket='mybucket', Key=content['Key'])
print(content['Key'], obj_dict['LastModified'])
Note: this client-level code is limited to listing at most 1000 objects. You would have to use a paginator, or implement your own loop, calling list_objects_v2() repeatedly with a continuation marker if there were more than 1000 objects.
OK, so that’s the low-level Client interface. Now onto the higher-level (more abstract) Resource interface.
Resource:
- this is the newer boto3 API abstraction
- it provides a high-level, object-oriented API
- it does not provide 100% API coverage of AWS services
- it uses identifiers and attributes
- it has actions (operations on resources)
- it exposes sub-resources and collections of AWS resources
- typically yields marshalled data, not primitive AWS data (e.g. DynamoDB attributes are native Python values representing primitive DynamoDB values)
- does result pagination for you
- it is generated from an AWS resource description
Here’s the equivalent example using resource-level access to an S3 bucket’s objects:
import boto3
s3 = boto3.resource('s3')
bucket = s3.Bucket('mybucket')
for obj in bucket.objects.all():
print(obj.key, obj.last_modified)
Note: in this case you do not have to make a second API call to get the objects; they’re available to you as a collection on the bucket. These collections of sub-resources are lazily-loaded.
You can see that the Resource version of the code is much simpler, more compact, and has more capability (for example it does pagination for you and it exposes properties instead of a raw dictionary). The Client version of the code would actually be more complicated than shown above if you wanted to include pagination.
Finally, onto Session which is fundamental to both Client and Resource and how both get access to AWS credentials, for example.
Session:
- stores configuration information (primarily credentials and selected region)
- allows you to create service clients and resources
- boto3 creates a default session for you when needed
A useful resource to learn more about these boto3 concepts is the introductory re:Invent video.
Update Jan 2023: per the Resources page in the boto3 documentation:
The AWS Python SDK team does not intend to add new features to the resources interface in boto3. Existing interfaces will continue to operate during boto3’s lifecycle. Customers can find access to newer service features through the client interface.
You can read more about the plans to deprecate resources at boto3/discussions/3563.
I am using Python 2.7.12 in Ubuntu 16.04 LTS. I’m learning how to use boto3 from the following link: https://boto3.readthedocs.io/en/latest/guide/quickstart.html#using-boto-3. My doubt is when to use resource, client, or session, and their respective functionality.
I’ll try and explain it as simple as possible. So there is no guarantee of the accuracy of the actual terms.
Session is where to initiate the connectivity to AWS services. E.g. following is default session that uses the default credential profile(e.g. ~/.aws/credentials, or assume your EC2 using IAM instance profile )
sqs = boto3.client('sqs')
s3 = boto3.resource('s3')
Because default session is limit to the profile or instance profile used, sometimes you need to use the custom session to override the default session configuration (e.g. region_name, endpoint_url, etc. ) e.g.
# custom resource session must use boto3.Session to do the override
my_west_session = boto3.Session(region_name = 'us-west-2')
my_east_session = boto3.Session(region_name = 'us-east-1')
backup_s3 = my_west_session.resource('s3')
video_s3 = my_east_session.resource('s3')
# you have two choices of create custom client session.
backup_s3c = my_west_session.client('s3')
video_s3c = boto3.client("s3", region_name = 'us-east-1')
Resource : This is the high-level service class recommended to be used. This allows you to tied particular AWS resources and passes it along, so you just use this abstraction than worry which target services are pointed to. As you notice from the session part, if you have a custom session, you just pass this abstract object than worrying about all custom regions,etc to pass along. Following is a complicated example
E.g.
import boto3
my_west_session = boto3.Session(region_name = 'us-west-2')
my_east_session = boto3.Session(region_name = 'us-east-1')
backup_s3 = my_west_session.resource("s3")
video_s3 = my_east_session.resource("s3")
backup_bucket = backup_s3.Bucket('backupbucket')
video_bucket = video_s3.Bucket('videobucket')
# just pass the instantiated bucket object
def list_bucket_contents(bucket):
for object in bucket.objects.all():
print(object.key)
list_bucket_contents(backup_bucket)
list_bucket_contents(video_bucket)
Client is a low level class object. For each client call, you need to explicitly specify the targeting resources, the designated service target name must be pass long. You will lose the abstraction ability.
For example, if you only deal with the default session, this looks similar to boto3.resource.
import boto3
s3 = boto3.client('s3')
def list_bucket_contents(bucket_name):
for object in s3.list_objects_v2(Bucket=bucket_name) :
print(object.key)
list_bucket_contents('Mybucket')
However, if you want to list objects from a bucket in different regions, you need to specify the explicit bucket parameter required for the client.
import boto3
backup_s3 = my_west_session.client('s3',region_name = 'us-west-2')
video_s3 = my_east_session.client('s3',region_name = 'us-east-1')
# you must pass boto3.Session.client and the bucket name
def list_bucket_contents(s3session, bucket_name):
response = s3session.list_objects_v2(Bucket=bucket_name)
if 'Contents' in response:
for obj in response['Contents']:
print(obj['key'])
list_bucket_contents(backup_s3, 'backupbucket')
list_bucket_contents(video_s3 , 'videobucket')
Client and Resource are two different abstractions within the boto3 SDK for making AWS service requests. If you want to make API calls to an AWS service with boto3, then you do so via a Client or a Resource.
You would typically choose to use either the Client abstraction or the Resource abstraction, but you can use both, as needed. I’ve outlined the differences below to help readers decide which to use.
Session is largely orthogonal to the concepts of Client and Resource (but is used by both).
Here’s some more detailed information on what Client, Resource, and Session are all about.
Client:
- this is the original boto3 API abstraction
- it provides low-level AWS service access
- all AWS service operations are supported by clients
- it exposes botocore client to the developer
- it typically maps 1:1 with the AWS service API
- it exposes snake-cased method names (e.g. ListBuckets API => list_buckets method)
- typically yields primitive, non-marshalled data (e.g. DynamoDB attributes are dicts representing primitive DynamoDB values)
- requires you to code result pagination
- it is generated from an AWS service description
Here’s an example of client-level access to an S3 bucket’s objects:
import boto3
client = boto3.client('s3')
response = client.list_objects_v2(Bucket='mybucket')
for content in response['Contents']:
obj_dict = client.get_object(Bucket='mybucket', Key=content['Key'])
print(content['Key'], obj_dict['LastModified'])
Note: this client-level code is limited to listing at most 1000 objects. You would have to use a paginator, or implement your own loop, calling list_objects_v2() repeatedly with a continuation marker if there were more than 1000 objects.
OK, so that’s the low-level Client interface. Now onto the higher-level (more abstract) Resource interface.
Resource:
- this is the newer boto3 API abstraction
- it provides a high-level, object-oriented API
- it does not provide 100% API coverage of AWS services
- it uses identifiers and attributes
- it has actions (operations on resources)
- it exposes sub-resources and collections of AWS resources
- typically yields marshalled data, not primitive AWS data (e.g. DynamoDB attributes are native Python values representing primitive DynamoDB values)
- does result pagination for you
- it is generated from an AWS resource description
Here’s the equivalent example using resource-level access to an S3 bucket’s objects:
import boto3
s3 = boto3.resource('s3')
bucket = s3.Bucket('mybucket')
for obj in bucket.objects.all():
print(obj.key, obj.last_modified)
Note: in this case you do not have to make a second API call to get the objects; they’re available to you as a collection on the bucket. These collections of sub-resources are lazily-loaded.
You can see that the Resource version of the code is much simpler, more compact, and has more capability (for example it does pagination for you and it exposes properties instead of a raw dictionary). The Client version of the code would actually be more complicated than shown above if you wanted to include pagination.
Finally, onto Session which is fundamental to both Client and Resource and how both get access to AWS credentials, for example.
Session:
- stores configuration information (primarily credentials and selected region)
- allows you to create service clients and resources
- boto3 creates a default session for you when needed
A useful resource to learn more about these boto3 concepts is the introductory re:Invent video.
Update Jan 2023: per the Resources page in the boto3 documentation:
The AWS Python SDK team does not intend to add new features to the resources interface in boto3. Existing interfaces will continue to operate during boto3’s lifecycle. Customers can find access to newer service features through the client interface.
You can read more about the plans to deprecate resources at boto3/discussions/3563.