aiohttp: Getting a server's response when the request status code is not 2XX

Question

I’m using aiohttp for async http requests, and I can’t figure out how to get the response from a server when the request returns a 4XX error.

    async def login(self, username: str, password: str) -> None:
        ...
        async with aiohttp.ClientSession(headers=self._headers) as session:
            async with session.post(route, data=data, headers=self._headers) as resp:
                if resp.ok:
                    response = await resp.json()
                    self._headers['Authorization'] = 'Bearer ' + response['access_token']
                else:
                    response = await resp.json()
                    raise InvalidGrant(response)

Using resp.json() works just fine if the response returns a 2XX code, however when it returns a 4XX error (in this case 400), it raises a aiohttp.client_exceptions.ClientConnectionError and doesn’t let me get the response that the server sent (which I need, since the server returns some sort of error message which is more descriptive than Bad Request). Is there no way to get the response with aiohttp if the request isn’t a success?

Asked By: professional8450

||

Source

Answer 1

The reason this issue is occuring is because of a side effect with response.ok. In older versions of aiohttp (3.7 and below), response.ok called response.raise_for_status(), which closed the TCP session and resulted in not being able to read the server’s response anymore.

To fix this, you simply need to move response = await resp.json() above the response.ok line, that way you save the response beforehand. For example:

    async def login(self, username: str, password: str) -> None:
        ...
        async with aiohttp.ClientSession(headers=self._headers) as session:
            async with session.post(route, data=data, headers=self._headers) as resp:
                response = await resp.json()
                if resp.ok:
                    self._headers['Authorization'] = 'Bearer ' + response['access_token']
                else:
                    raise InvalidGrant(response)

This issue has been fixed in aiohttp 3.8 however: https://github.com/aio-libs/aiohttp/pull/5404

Answered By: professional8450

Answer 2

Don’t use raise_for_status=True as a keyword argument anywhere. Instead manually call response.raise_for_status() after you retrieve the response, its status code, message etc

async def fetch_feed_items(
    feed: Feed, request_headers: CIMultiDict, session: aiohttp.ClientSession
) -> Tuple[Feed, CIMultiDictProxy, int, str]:
    """Load data from url.

    Args:
        feed (Feed): object which contains url, etag, last_modified and feed_id
        request_headers (CIMultiDict): headers sent with the request
        session (aiohttp.ClientSession): an aiohttp.ClientSession instance

    Returns:
        Tuple[Feed, CIMultiDictProxy, int, str]: Returns information about the source
    """
    response_headers = {}
    text = ""
    status = None
    try:
        async with session.get(feed.url, headers=request_headers, ssl=False) as response:
            text = await response.text()
            response_headers = response.headers
            status = response.status
            message = response.reason
            # Dont use raise_for_status=True option as it doesn't capture the status code and message
            response.raise_for_status()
    except aiohttp.ClientError as e:
        print(feed.url, "aiohttp client error", e)
    except aiohttp.http.HttpProcessingError as e:
        print(feed.url, "aiohttp processing error", e)
    except asyncio.TimeoutError as e:
        print(feed.url, "asyncio timeout error", e)
    except Exception as e:
        print(feed.url, "generic error", e)
    finally:
        return feed, response_headers, status, message, text

In the above snippet, when there is a 4xx or 5xx error, the response headers, status, message values are stored in a variable first and then response.raise_for_status triggers the exception

Answered By: PirateApp

aiohttp: Getting a server's response when the request status code is not 2XX

Question:

Answers: