how to get response_time and response_size while using aiohttp

Question

Is it possible to get response time and response size for each request made using aiohttp?

The documentation seems not to have those properties anywhere.

Thanks

Asked By: Dan

||

Source

Answer 1

One possibility might be:

measure point in time before request
measure point in time after request
the difference is the response time
with ‘response.text()’ you get the response and can determine the length with ‘len()’

A small self-contained example could look like this:

import time
import asyncio
from aiohttp import ClientSession


async def fetch(session, url):
    start = time.time()
    async with session.get(url) as response:
        result = await response.text()
        end = time.time()
        print(url, ": ", end - start, "response length:", len(result))
        return result


async def crawl(urls: set):
    async with ClientSession() as session:
        tasks = []
        for url in urls:
            tasks.append(
                fetch(session, url)
            )
        await asyncio.gather(*tasks)


if __name__ == "__main__":
    urlSet = {"https://www.software7.biz/tst/number.php",
              "https://www.software7.biz/tst/number1.php",
              "https://www.software7.biz"}
    asyncio.run(crawl(urlSet))

Test

The two endpoints number.php and number1.php have a delay on server side of 3 respective 1 second and are returning a two digit number each.

The output in the debug console looks like this then:

https://www.software7.biz :  0.16438698768615723 response length: 4431
https://www.software7.biz/tst/number1.php :  1.249755859375 response length: 2
https://www.software7.biz/tst/number.php :  3.214473009109497 response length: 2

Answered By: Stephan Schlecht

Answer 2

len(response.text()) will return size of decompressed response.
If you want the size of the raw compressed response you need to set auto_decompress=False during creation of aiohttp.ClientSession. After that you can get it with len(await response.read()).
But it’ll make response.text() unavailable since it needs uncompressed response. To make it available again you’ll have to decompress it manually:

import time
import zlib
import brotli

async with aiohttp.ClientSession(auto_decompress=False) as session:
    start = time.monotonic()
    response = await session.get(url='www.test.com')
    response_time = time.monotonic() - start
    response_size = len(await response.read())

    encoding = response.headers['Content-Encoding']
    if encoding == 'gzip':
        response._body = zlib.decompress(response._body, 16 + zlib.MAX_WBITS)
    elif encoding == 'deflate':
        response._body = zlib.decompress(response._body, -zlib.MAX_WBITS)
    elif encoding == 'br':
        response._body == brotli.decompress(response._body)

    response_text = await response.text()

About time.time() from pymotw.com:

Because time.time() looks at the system clock, and the system clock can be changed by the user or system services for synchronizing clocks across multiple computers, calling time.time() repeatedly may produce values that go forwards and backwards. This can result in unexpected behavior when trying to measure durations or otherwise use those times for computation. Avoid those situations by using time.monotonic(), which always returns values that go forward.

aiohttp docs suggest to use loop.time() (which is also monotonic):

async def on_request_start(session, trace_config_ctx, params):
    trace_config_ctx.start = asyncio.get_event_loop().time()

async def on_request_end(session, trace_config_ctx, params):
    elapsed = asyncio.get_event_loop().time() - trace_config_ctx.start
    print("Request took {}".format(elapsed))

trace_config = aiohttp.TraceConfig()
trace_config.on_request_start.append(on_request_start)
trace_config.on_request_end.append(on_request_end)
async with aiohttp.ClientSession(trace_configs=[trace_config]) as client:
    client.get('http://example.com/some/redirect/')

Answered By: Sergey Geron

Answer 3

You can get the size of the response content from the headers:

response.headers['content-length']

Answered By: Joe

how to get response_time and response_size while using aiohttp

Question:

Answers: