What is the difference between boto3 list_objects and list_objects_v2?

Question

I’m trying to list objects in an Amazon s3 bucket in python using boto3.

It seems boto3 has 2 functions for listing the objects in a bucket: list_objects() and list_objects_v2().

What is the difference between the 2 and what is the benefit of using one over the other?

Asked By: tamjd1

||

Source

Answer 1

Comparison side by side.

list_objects() :

response = client.list_objects(
    Bucket='string',
    Delimiter='string',
    EncodingType='url',
    #Marker to list continuous page
    Marker='string',
    MaxKeys=123,
    Prefix='string'
)

list_objects_v2()

response = client.list_objects_v2(
    Bucket='string',
    Delimiter='string',
    EncodingType='url',
    MaxKeys=123,
    Prefix='string',
    # Replace marker to list continuous page
    ContinuationToken='string',

    # set to True to fetch key owner info. Default is False.
    FetchOwner=True|False,

    # This is similar to the Marker in list_object()
    StartAfter='string'
)

Added features. Due to the 1000 keys per page listing limits, using marker to list multiple pages can be an headache. Logically, you need to keep track the last key you successfully processed. With ContinuationToken, you don’t need to know the last key, you just check existence of NextContinuationToken in the response. You can spawn parallel process to deal with multiply of 1000 keys without dealing with the last key to fetch next page.

Answered By: mootmoot

Answer 2

This answer elaborates upon the answer provided by @mootmoot: https://stackoverflow.com/a/37539994/9286461

Here is an example of how list_objects_v2() can avoid tracking the previous token like this:

bkt_cur = '[BUCKET_NAME]'
s3pfx_cur = '[some_s3_prefix]'

lst = []
paginator = s3.get_paginator("list_objects_v2")
for page in paginator.paginate(**{'Bucket' : bkt_cur, 'Prefix': s3pfx_cur}):
    try:
        contents = page["Contents"]
        lst.extend(contents)
    except KeyError:
        break

Answered By: Ryan Y

What is the difference between boto3 list_objects and list_objects_v2?

Question:

Answers: