how to implement exponential backoff in amazon sqs without writing code

Question

I have a simple task that requires a 3rd party.
When a request comes, I push it to an amazon sqs queue, pull it in a worker and call the 3rd party. In case of a time out, I want to implement an exponential backoff (try again in 2 secs, then 4 then 8, then…)
with a max retry.

Using python, boto -> sqs

I’ve looked for built in parameters to allow me to do so with as little code as possible (ideally, no code at all).

Something like

from boto import sqs

def handle_message(message):
    try:
      # send a post to api
    except TimeOut, err:
      # send_back_to_itself in 2/4/8 sec
      if delay < DELAY_LIMIT:
          queue.write(message, delay=secs)

Asked By: WebQube

||

Source

Answer 1

According to the AWS SDK docs, you get exponential backoff for free if you’re using one of the official SDK libraries. So it seems all you need to do is set your max_attempts count, and everything after the first attempt will back off exponentially:

import boto3
from botocore.config import Config

config = Config(
    retries = dict(
        max_attempts = 10   # docs say default is 5
    )
)

sqs = boto3.client('sqs', config=config)
sqs.receive_message(QueueUrl='https://sqs.us-east-1.amazonaws.com/<your-account-num>/foobar')

Answered By: Matt Morgan

Answer 2

I don’t think "no code at all" is possible as of summer 2021. But there is a great blog post on how to do this with coding examples https://aws.amazon.com/blogs/compute/using-amazon-sqs-dead-letter-queues-to-replay-messages/

Code sample from the link above:

def handler(event, context):
    """Lambda function handler."""
    for record in event['Records']:
        nbReplay = 0
        # number of replay
        if 'sqs-dlq-replay-nb' in record['messageAttributes']:
            nbReplay = int(record['messageAttributes']['sqs-dlq-replay-nb']["stringValue"])

        nbReplay += 1
        if nbReplay > config.MAX_ATTEMPS:
            raise MaxAttempsError(replay=nbReplay, max=config.MAX_ATTEMPS)

        # SQS attributes
        attributes = record['messageAttributes']
        attributes.update({'sqs-dlq-replay-nb': {'StringValue': str(nbReplay), 'DataType': 'Number'}})

        _sqs_attributes_cleaner(attributes)

        # Backoff
        b = backoff.ExpoBackoffFullJitter(base=config.BACKOFF_RATE, cap=config.MESSAGE_RETENTION_PERIOD)
        delaySeconds = b.backoff(n=int(nbReplay))

        # SQS
        SQS.send_message(
            QueueUrl=config.SQS_MAIN_URL,
            MessageBody=record['body'],
            DelaySeconds=int(delaySeconds),
            MessageAttributes=record['messageAttributes']
        )

Answered By: Sergey Yarotskiy

how to implement exponential backoff in amazon sqs without writing code

Question:

Answers: