Pagination in boto3 ec2 describe instance

Question

I am having an issue with pagination in boto3 & not getting all instances in the aws account.

Only getting 50% of the instances with below (around 2000 where as there are 4000)

Below is my code

import boto3

ec2 = boto3.client('ec2')

paginator = ec2.get_paginator('describe_instances')
response = paginator.paginate().build_full_result()

ec2_instance = response['Reservations']


for instance in ec2_instance:
    print(instance['Instances'][0]['InstanceId'])

Asked By: rkj

||

Source

Answer 1

The response from describe_instances() is:

{
    'Reservations': [
        {
            'Groups': [
                {
                    'GroupName': 'string',
                    'GroupId': 'string'
                },
            ],
            'Instances': [
                {
                    'AmiLaunchIndex': 123,
 ...

Notice that the response is:

A dictionary
Where Reservations is a list containing:
- Instances, which is a list

Therefore, the code really needs to loop through all Reservations and instances.

At the moment, your code is looping through the Reservations (incorrectly calling them instances), and is then only retrieving the first ([0]) instance from that Reservation.

You’ll probably want some code like this:

for reservation in response['Reservations']:
  for instance in reservation['Instances']:
    print(instance['InstanceId'])

Answered By: John Rotenstein

Answer 2

If you use resource not client then you don’t need to worry about pagination.

import boto3
ec2 = boto3.resource('ec2')

response = ec2.instances.all()
for item in response:
    print(item.id)

The code will give you all results.

Answered By: Lamanus

Answer 3

Instead of calling build_full_result(), you could do this:

from boto3 import client

for page in client('ec2').get_paginator('describe_instances').paginate():
    for res in page['Reservations']:
        for inst in res['Instances']:
            print( inst['InstanceId'])

Calling build_full_result() could defeat the purpose of pagination, namely to manage memory.

Answered By: Brian Fitzgerald

Pagination in boto3 ec2 describe instance

Question:

Answers: