How do you use "NextToken" in AWS API calls
Question:
I’ve run into a little issue that I am really struggling to understand how it works. I have a tool I am writing that basically does a describe-organization to collect all the accounts in our AWS organization. Per the documentation here it says it responds with a json of the accounts which in my case will be hundreds and hundreds of accounts. So I wrote some very simple code to switch roles into our master account and make the call:
import boto3
import uuid
import pprint
iam_client = boto3.client('iam')
sts_client = boto3.client('sts')
org_client = boto3.client('organizations')
print("Starting in account: %s" % sts_client.get_caller_identity().get('Account'))
assumedRoleObject = sts_client.assume_role(
RoleArn="arn:aws:iam::123456xxx:role/MsCrossAccountAccessRole",
RoleSessionName="MasterPayer"
)
credentials = assumedRoleObject['Credentials']
org_client = boto3.client(
'organizations',
aws_access_key_id = credentials['AccessKeyId'],
aws_secret_access_key = credentials['SecretAccessKey'],
aws_session_token = credentials['SessionToken'],
)
getListAccounts = org_client.list_accounts(
NextToken='string'
)
But when I execute the code, I get the following error:
“botocore.errorfactory.InvalidInputException: An error occurred (InvalidInputException) when calling the ListAccounts operation: You specified an invalid value for nextToken. You must get the value from the response to a previous call to the API.”
I’m really stumped on what that means. I see the NextToken, and I can find many references to it in the AWS documentation but I can’t figure out how to actually USE it. Like, what do I need to do with it?
Answers:
Don’t take the boto3 examples literally (they are not actual examples). Here is how this works:
1) The first time you make a call to list_accounts you’ll do it without the NextToken, so simply
getListAccounts = org_client.list_accounts()
2) This will return a JSON response which looks roughly like this (this is what is saved in your getListAccounts variable):
{
"Accounts": [<lots of accounts information>],
"NextToken": <some token>
}
Note that the NextToken is only returned in case you have more accounts than one list_accounts call can return, usually this is 100 (the boto3 documentation does not state how many by default). If all accounts were returned in one call there is no NextToken in the response!
3) So if and only if not all accounts were returned in the first call you now want to return more accounts and you will have to use the NextToken in order to do this:
getListAccountsMore = org_client.list_accounts(NextToken=getListAccounts['NextToken'])
4) Repeat until no NextToken is returned in the response anymore (then you retrieved all accounts).
This is how the AWS SDK handles pagination in many cases. You will see the usage of the NextToken in other service clients as well.
Here my example where I used NextToken to check if a secret exist inside the SecretManager.
There are also some print useful to visualize the first time
def check_if_secret_existv2(username):
results_for_call=5
response = client.list_secrets(MaxResults=results_for_call)
i=0
while True:
i=i+1
if 'NextToken' in response:
response = client.list_secrets(MaxResults=results_for_call,NextToken=response['NextToken'])
else:
response = client.list_secrets(MaxResults=results_for_call)
for secret in response['SecretList']:
print(secret['Name'])
if secret['Name'] == username:
return True
print('End cycle '+str(i))
if 'NextToken' not in response:
break
return False
print(check_if_secret_existv2(myusername))
Instead you can use get_paginator api.
find below example, In my use case i had to get all the values of SSM parameter store and wanted to compare it with a string.
import boto3
import sys
LBURL = sys.argv[1].strip()
client = boto3.client('ssm')
p = client.get_paginator('describe_parameters')
paginator = p.paginate().build_full_result()
for page in paginator['Parameters']:
response = client.get_parameter(Name=page['Name'])
value = response['Parameter']['Value']
if LBURL in value:
print("Name is: " + page['Name'] + " and Value is: " + value)
I know it’s a different client but for anyone still having issues this took me some time to figure out:
This client required the initial request params also be included. Here’s the code:
import boto3
lex_client = boto3.client("lex-models")
response = lex_client.get_slot_types(
nameContains='search_string',
maxResults=50
)
slot_types = response['slotTypes']
while 'nextToken' in response.keys():
response = lex_client.get_slot_types(nameContains='search_string', maxResults=50, nextToken=response['nextToken'])
slot_types.extend(response['slotTypes'])
print('here are the slot types', slot_types)
Same as other answer, but with a short snippet with a simple while loop.
response = client.list_accounts()
results = response["Accounts"]
while "NextToken" in response:
response = client.list_accounts(NextToken=response["NextToken"])
results.extend(response["Accounts"])
I tried to list secrets names in my secrets manager using boto3 python:
secrets = secret_client.list_secrets()
secrets_manager = (secrets['SecretList'])
for secret in secrets_manager:
print ("{0}".format(secret['Name']))
The complete list was around 20, but the output was only 5 secret names.
Updated the code to below, it worked:
secrets = secret_client.list_secrets()
secrets_manager = (secrets['SecretList'])
while "NextToken" in secrets:
secrets = secret_client.list_secrets(NextToken=secrets["NextToken"])
secrets_manager.extend(secrets['SecretList'])
for secret in secrets_manager:
print ("{0}".format(secret['Name']))
And another example to return a list of LogGroupNames using nextToken:
response = logs_client.describe_log_groups(limit=50)
results = response["logGroups"]
log.info(f' response: {response}')
log.info(f' results: {results}')
while "nextToken" in response:
log.info(' nextToken')
response = logs_client.describe_log_groups(nextToken=response["nextToken"], limit=50)
results.extend(response["logGroups"])
log_group_name_list = []
for groups in results:
log_group_name = groups['logGroupName']
log_group_name_list.append(log_group_name)
log.info(f' log_group_name_list: {log_group_name_list}')
log.info(f' log_group_name_list:type: {type(log_group_name_list)}')
resp = dynamodb.execute_statement(Statement="SELECT gateway FROM
Table_Name")
data = resp['Items']
#print(resp['NextToken'])
while 'NextToken' in resp:
resp = dynamodb.execute_statement(Statement="SELECT gateway FROM
Table_Name", NextToken =resp['NextToken'])
data.extend(resp['Items'])
I’ve run into a little issue that I am really struggling to understand how it works. I have a tool I am writing that basically does a describe-organization to collect all the accounts in our AWS organization. Per the documentation here it says it responds with a json of the accounts which in my case will be hundreds and hundreds of accounts. So I wrote some very simple code to switch roles into our master account and make the call:
import boto3
import uuid
import pprint
iam_client = boto3.client('iam')
sts_client = boto3.client('sts')
org_client = boto3.client('organizations')
print("Starting in account: %s" % sts_client.get_caller_identity().get('Account'))
assumedRoleObject = sts_client.assume_role(
RoleArn="arn:aws:iam::123456xxx:role/MsCrossAccountAccessRole",
RoleSessionName="MasterPayer"
)
credentials = assumedRoleObject['Credentials']
org_client = boto3.client(
'organizations',
aws_access_key_id = credentials['AccessKeyId'],
aws_secret_access_key = credentials['SecretAccessKey'],
aws_session_token = credentials['SessionToken'],
)
getListAccounts = org_client.list_accounts(
NextToken='string'
)
But when I execute the code, I get the following error:
“botocore.errorfactory.InvalidInputException: An error occurred (InvalidInputException) when calling the ListAccounts operation: You specified an invalid value for nextToken. You must get the value from the response to a previous call to the API.”
I’m really stumped on what that means. I see the NextToken, and I can find many references to it in the AWS documentation but I can’t figure out how to actually USE it. Like, what do I need to do with it?
Don’t take the boto3 examples literally (they are not actual examples). Here is how this works:
1) The first time you make a call to list_accounts you’ll do it without the NextToken, so simply
getListAccounts = org_client.list_accounts()
2) This will return a JSON response which looks roughly like this (this is what is saved in your getListAccounts variable):
{
"Accounts": [<lots of accounts information>],
"NextToken": <some token>
}
Note that the NextToken is only returned in case you have more accounts than one list_accounts call can return, usually this is 100 (the boto3 documentation does not state how many by default). If all accounts were returned in one call there is no NextToken in the response!
3) So if and only if not all accounts were returned in the first call you now want to return more accounts and you will have to use the NextToken in order to do this:
getListAccountsMore = org_client.list_accounts(NextToken=getListAccounts['NextToken'])
4) Repeat until no NextToken is returned in the response anymore (then you retrieved all accounts).
This is how the AWS SDK handles pagination in many cases. You will see the usage of the NextToken in other service clients as well.
Here my example where I used NextToken to check if a secret exist inside the SecretManager.
There are also some print useful to visualize the first time
def check_if_secret_existv2(username):
results_for_call=5
response = client.list_secrets(MaxResults=results_for_call)
i=0
while True:
i=i+1
if 'NextToken' in response:
response = client.list_secrets(MaxResults=results_for_call,NextToken=response['NextToken'])
else:
response = client.list_secrets(MaxResults=results_for_call)
for secret in response['SecretList']:
print(secret['Name'])
if secret['Name'] == username:
return True
print('End cycle '+str(i))
if 'NextToken' not in response:
break
return False
print(check_if_secret_existv2(myusername))
Instead you can use get_paginator api.
find below example, In my use case i had to get all the values of SSM parameter store and wanted to compare it with a string.
import boto3
import sys
LBURL = sys.argv[1].strip()
client = boto3.client('ssm')
p = client.get_paginator('describe_parameters')
paginator = p.paginate().build_full_result()
for page in paginator['Parameters']:
response = client.get_parameter(Name=page['Name'])
value = response['Parameter']['Value']
if LBURL in value:
print("Name is: " + page['Name'] + " and Value is: " + value)
I know it’s a different client but for anyone still having issues this took me some time to figure out:
This client required the initial request params also be included. Here’s the code:
import boto3
lex_client = boto3.client("lex-models")
response = lex_client.get_slot_types(
nameContains='search_string',
maxResults=50
)
slot_types = response['slotTypes']
while 'nextToken' in response.keys():
response = lex_client.get_slot_types(nameContains='search_string', maxResults=50, nextToken=response['nextToken'])
slot_types.extend(response['slotTypes'])
print('here are the slot types', slot_types)
Same as other answer, but with a short snippet with a simple while loop.
response = client.list_accounts()
results = response["Accounts"]
while "NextToken" in response:
response = client.list_accounts(NextToken=response["NextToken"])
results.extend(response["Accounts"])
I tried to list secrets names in my secrets manager using boto3 python:
secrets = secret_client.list_secrets()
secrets_manager = (secrets['SecretList'])
for secret in secrets_manager:
print ("{0}".format(secret['Name']))
The complete list was around 20, but the output was only 5 secret names.
Updated the code to below, it worked:
secrets = secret_client.list_secrets()
secrets_manager = (secrets['SecretList'])
while "NextToken" in secrets:
secrets = secret_client.list_secrets(NextToken=secrets["NextToken"])
secrets_manager.extend(secrets['SecretList'])
for secret in secrets_manager:
print ("{0}".format(secret['Name']))
And another example to return a list of LogGroupNames using nextToken:
response = logs_client.describe_log_groups(limit=50)
results = response["logGroups"]
log.info(f' response: {response}')
log.info(f' results: {results}')
while "nextToken" in response:
log.info(' nextToken')
response = logs_client.describe_log_groups(nextToken=response["nextToken"], limit=50)
results.extend(response["logGroups"])
log_group_name_list = []
for groups in results:
log_group_name = groups['logGroupName']
log_group_name_list.append(log_group_name)
log.info(f' log_group_name_list: {log_group_name_list}')
log.info(f' log_group_name_list:type: {type(log_group_name_list)}')
resp = dynamodb.execute_statement(Statement="SELECT gateway FROM
Table_Name")
data = resp['Items']
#print(resp['NextToken'])
while 'NextToken' in resp:
resp = dynamodb.execute_statement(Statement="SELECT gateway FROM
Table_Name", NextToken =resp['NextToken'])
data.extend(resp['Items'])