AWS lambda with python asyncio. Event loop closed problem?

Question:

Closing the event loop in aws lambda affects future lambda runs??

  • I have some aysncio python code running within an aws lambda service. The logic of the code is as follows

    def lambda_handler(event,context):
    loop = asyncio.get_event_loop()
    # perform all operations with the loop
    loop.close()
    return results

  • If I run this once, it appears to work fine. However, if I rerun it immediately afterwards, I get an error saying Event loop closed

  • Why is this happening? Shouldn’t each lambda run be independent of the last? After all lambda is supposed to be stateless
Asked By: Arran Duff

||
Source

Answers:

Add this line of the top of the code,

asyncio.set_event_loop(asyncio.new_event_loop())

so that it is global.

Alternatively,

Replace,

loop = asyncio.get_event_loop()

with,

loop = asyncio.new_event_loop()

It is happening since you already closed the loop.

That should work.

Answered By: Kannaiyan

After all lambda is supposed to be stateless

Your function is supposed to be stateless.

From https://aws.amazon.com/lambda/faqs/:

Q: What is an AWS Lambda function?

…

The code must be written in a “stateless” style i.e. it should assume there is no affinity to the underlying compute infrastructure.

…

Q: Will AWS Lambda reuse function instances?

To improve performance, AWS Lambda may choose to retain an instance of your function and reuse it to serve a subsequent request, rather than creating a new copy. To learn more about how Lambda reuses function instances, visit our documentation. Your code should not assume that this will always happen.

The current python instance is reused for performance reasons, but its reuse or non-reuse must never be relied on. So while AWS Lambda is not itself always stateless, your programming methodology should be. Hopefully that clears up your confusion as to why that is happening!

Answered By: ParkerD

Python 3.7+

You can use the higher level asyncio.run() that will take care of things.

def handler(event, context):
    asyncio.run(main())

async def main():
    # your async code here

This will close the loop at the end and open a new one when running lambda again. .run() is also recommended by asyncio maintainers.

Answered By: Anatol Bivol

As of 2023-1-3, I found all existing solutions don’t work within Python3.9 environment:

  • new a loop: got Future <Future pending> attached to a different loop
  • asyncio.run: Asyncio Event Loop is Closed
  • and some other tricks seen in other answers I didn’t record

I have to do this:

import asyncio

async def work(msg):
    await do_something_with(msg)

def lambda_handler(event, context):
    msg = event.get("body", "emptybody") # your msg should vary
    loop = asyncio.get_event_loop()
    if loop.is_closed(): # I don't know why
        loop = asyncio.new_event_loop()
    loop.run_until_complete(work(msg))

Only then the first run and consequent runs won’t fail.

My best random guess is that Lambda sometimes starts from the top of the code, and sometimes just reuses lambda_handler, and it will close event loops sometimes

Answered By: SnoopyGuo