Asynchronous Python – fire and forget HTTP request

Question

Not sure if that’s achievable. I want to fire an HTTP POST request from a script, but not wait for a response. Instead I want to return immediately.

I tries something along these lines:

#!/usr/bin/env python3

import asyncio
import aiohttp

async def fire():
    await client.post('http://httpresponder.com/xyz')

async def main():
    asyncio.ensure_future(fire())

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    client = aiohttp.ClientSession(loop=loop)
    loop.run_until_complete(main())

The script returns immediately without errors, but the HTTP request never arrives at the destination. Can I fire the POST request, but not wait for response from the server, just terminate the moment the request is sent?

Asked By: luqo33

||

Source

Answer 1

I have answered a rather similar question.

async def main():
    asyncio.ensure_future(fire())

ensure_future schedules coro execution, but does not wait for its completion and run_until_complete does not wait for the completion of all futures.

This should fix it:

async def main():
    await fire()

Answered By: Artemiy Rodionov

Answer 2

Using asyncio you can write a simple decorator as @background. Now you can write whatever inside foo() and the control-flow will not wait for its completion.

import asyncio
import time


def background(f):
    from functools import wraps
    @wraps(f)
    def wrapped(*args, **kwargs):
        loop = asyncio.get_event_loop()
        if callable(f):
            return loop.run_in_executor(None, f, *args, **kwargs)
        else:
            raise TypeError('Task must be a callable')    
    return wrapped


@background
def foo():
    time.sleep(1)
    print("foo() completed")


print("Hello")
foo()
print("I didn't wait for foo()")

Produces:

Hello
I didn't wait for foo()
foo() completed

Answered By: nehem

Answer 3

I noticed that the solutions offered by others weren’t satisfying your initial question, so I came up with a simple and synchronous solution for you, there is no need for fancy multithreading. This problem can be solved by using a lower-level library such as socket that comes built-in with python. The idea is sending the get request and not caring whether the other party returns a 200 or 400+, thus allowing us to truly fire and forget, not wasting any precious time, not awaiting anything. Here is the full code demonstrating the usefulness and implementation

and the general idea is doing

write_only_client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
write_only_client.connect((target_host, target_port))
write_only_client.send(request.encode())
write_only_client.close()

notice how we don’t do any write_only_client.recv calls, which is exactly why we are able to do a true fire and forget.

P.S.
Although it is not practiced in my code example, it is best to keep the socket open and have it as a global variable to avoid it being garbage collected during a scope change, as closing the socket too soon might cause problems (it did cause problems when I closed an SSL connection to an API).

Answered By: Movsisyan

Answer 4

It’s a bit verbose, but you can use the standard library http.client.HTTPSConnection:

import http.client, json
from datetime import datetime

d = datetime.now()
conn = http.client.HTTPSConnection("medium.com")

#not needed for this call, but you may need to send data on your calls
payload = json.dumps({"Some": [{"json": ["string"]}]})

#not needed for this call, but you may need to send headers on your calls
headers = {
  'Content-Type': 'application/json'
}

# add querystring params here if needed
urlPath = "/towards-data-science/current-time-python-4417c0f3bc4f"

# do the call
conn.request("GET", urlPath, payload, headers)
print('Needed time to do the call: ', datetime.now() - d)
d = datetime.now()

#You can remove code from this point on and call is still made
res = conn.getresponse()
print('Time getting response (could skip): ', datetime.now() - d)
d = datetime.now()

data = res.read()
print('Time to read: ', datetime.now() - d)
# print(data.decode("utf-8"))

>>> Needed time to do the call:  0:00:00.180057
>>> Time getting response (could skip):  0:00:00.646882
>>> Time to read:  0:00:00.001001

Good luck!

Answered By: Marquinho Peli

Asynchronous Python – fire and forget HTTP request

Question:

Answers: