How to send multiple batch messages from a list to an SQS queue using Boto3 and Python

Question

I have a list of integers (integerList) that I’d like to pass into an SQS queue where each message into the queue is an integer from the list.

I can do this one message at a time with the send_message() command, and the code for that is below.

import boto3

sqsResource = boto3.resource('sqs')

def write_sqs(integerList):
    queue = sqsResource.get_queue_by_name(QueueName=NAMEOFQUEUEHERE)
    for i in integerList:
        response = queue.send_message(MessageBody=str(i),
                                      MessageGroupId='TESTING')

However, I’d like to speed up the function and send the messages in batches. Currently, AWS SQS allows batching up to 10 messages at a time with the send_messages() command, but I’m not sure how to build the Entries= attribute for the batch send. I’m breaking down the integerList into smaller lists of 10 using chunks = [integerList[x:x+10] for x in range(0, len(integerList), 10)], but the next steps are unclear.

Asked By: grove80904

||

Source

Answer 1

According to the documentation Entries is a list of messages.
For each entry in Entries the parameters type are detailed at the link.

import boto3

sqsResource = boto3.resource('sqs')

def write_sqs(integerList):
    queue = sqsResource.get_queue_by_name(QueueName=NAMEOFQUEUEHERE)
    entries = []

    for i in integerList:
        entry =  {
            'Id': 'id%s' % str(integerList[i]),
            'MessageBody': str(integerList[i])
            }
        entries.append(entry)

    response = queue.send_messages(entries)

Answered By: liorko

Answer 2

Building on the answer from @liorko and some trial and error, this seems to work and is much faster than the 1-by-1 method I was using before.

import boto3

sqsResource = boto3.resource('sqs')

def write_sqs(integerList):
    queue = sqsResource.get_queue_by_name(QueueName=NAMEOFQUEUEHERE)
    maxBatchSize = 10 #current maximum allowed
    chunks = [integerList[x:x+maxBatchSize] for x in range(0, len(integerList), maxBatchSize)]
    for chunk in chunks:
        entries = []
        for x in chunk:
            entry = {'Id': str(x), 
                     'MessageBody': str(x), 
                     'MessageGroupId': 'ANYTHINGYOUWANT'}
            entries.append(entry)
        response = queue.send_messages(Entries=entries)

Answered By: grove80904

Answer 3

According to the new version of boto3, the method to use is send_message_batch, here is the updated code:

import boto3

sqs_client = boto3.client("sqs", region_name="eu-west-1")

entries = []
for i in integer_list:
    entry =  {
        'Id': 'id%s' % str(integerList[i]),
        'MessageBody': str(integerList[i])
        }
    entries.append(entry)
    response = queue.send_message_batch(Entries=entries, QueueUrl="https://sqs.eu-west-1.amazonaws.com/000000000000/my-queue-name")

Source: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs.html#SQS.Client.send_message_batch

Answered By: Sergio Lema

How to send multiple batch messages from a list to an SQS queue using Boto3 and Python

Question:

Answers: