retry with python requests when status_code = 200

Question

The API I’m sending requests to has a bit of an unusual format for its responses

It always returns status_code = 200
There’s an additional error key inside the returned json that details the actual status of the response:

2.1. error = 0 means it successfully completes

2.2. error != 0 means something went wrong

I’m trying use the Retry class in urlib3, but so far I understand it only uses the status_code from the response, not its actual content.

Are there any other options?

Asked By: caverac

||

Source

Answer 1

If I’m hearing you right, then there are two cases in which you have ‘errors’ to handle:

Any non-200 response from the web server (i.e. 500, 403, etc.)
whenever the API returns a non-zero value for ‘error’ in the JSON response as the server always responds with an HTTP 200 even if your request is bad.

Given that we need to handle two completely different cases which trigger a retry, it’d be easier to write your own retry handler rather than trying to hack our way into this with the urllib3 library or similar, as we can specifically specify the cases where we need to do a retry.

You might try something like this approach, which also takes into account the number of requests you’re making to determine if there’s a repeated error case, and in cases of API response errors or HTTP errors, we use an (suggested via comments on my initial answer) ‘exponential backoff’ approach to retries so you don’t constantly tax a server – this means that each successive retry has a different ‘sleep’ period before retrying, until we reach a MAX_RETRY count, as written it’s a base increment of 1 second for first retry attempt, 2 seconds for second retry, 4 seconds for third retry, etc. which will permit the server to catch up if it has to rather than just constantly over-tax the server.

import requests
import time

MAX_RETRY = 5


def make_request():
    '''This makes a single request to the server to get data from it.'''
    # Replace 'get' with whichever method you're using, and the URL with the actual API URL
    r = requests.get('http://api.example.com')
    
    # If r.status_code is not 200, treat it as an error.
    if r.status_code != 200:
        raise RuntimeError(f"HTTP Response Code {r.status_code} received from server."
    else:
        j = r.json()
        if j['error'] != 0:
            raise RuntimeError(f"API Error Code {j['error']} received from server."
        else:
            return j
            

def request_with_retry(backoff_in_seconds=1):
    '''This makes a request retry up to MAX_RETRY set above with exponential backoff.'''
    attempts = 1
    while True:
        try:
            data = make_request()
            return data
        except RuntimeError as err:
            print(err)
            if attempts > MAX_RETRY:
                raise RuntimeError("Maximum number of attempts exceeded, aborting.")
                
            sleep = backoff_in_seconds * 2 ** (attempts - 1)
            print(f"Retrying request (attempt #{attempts}) in {sleep} seconds...")
            time.sleep(sleep)
            attempts += 1

Then, you couple these two functions together with the following to actually attempt to get data from the API server and then either error hard or do something with it if there’s no errors encountered:

# This code actually *calls* these functions which contain the request with retry and 
# exponential backoff *and* the individual request process for a single request.
try:
    data = request_with_retry()
except RuntimeError as err:
    print(err)
    exit(1)

After that code, you can just ‘do something’ with data which is the JSON(?) output of your API, even if this part is included in another function. You just need the two dependent functions (done this way to reduce code duplication).

Answered By: Thomas Ward

retry with python requests when status_code = 200

Question:

Answers: