How to use await in a python lambda
Question:
I’m trying to do something like this:
mylist.sort(key=lambda x: await somefunction(x))
But I get this error:
SyntaxError: 'await' outside async function
Which makes sense because the lambda is not async.
I tried to use async lambda x: ... but that throws a SyntaxError: invalid syntax.
Pep 492 states:
Syntax for asynchronous lambda functions could be provided, but this construct is outside of the scope of this PEP.
But I could not find out if that syntax was implemented in CPython.
Is there a way to declare an async lambda, or to use an async function for sorting a list?
Answers:
You can’t. There is no async lambda, and even if there were, you coudln’t pass it in as key function to list.sort(), since a key function will be called as a synchronous function and not awaited. An easy work-around is to annotate your list yourself:
mylist_annotated = [(await some_function(x), x) for x in mylist]
mylist_annotated.sort()
mylist = [x for key, x in mylist_annotated]
Note that await expressions in list comprehensions are only supported in Python 3.6+. If you’re using 3.5, you can do the following:
mylist_annotated = []
for x in mylist:
mylist_annotated.append((await some_function(x), x))
mylist_annotated.sort()
mylist = [x for key, x in mylist_annotated]
The answer from Sven Marnach has an Edge case.
If you try and sort a list that has 2 items that produce the same search key but are different and are not directly sortable, it will crash.
mylist = [{'score':50,'name':'bob'},{'score':50,'name':'linda'}]
mylist_annotated = [(x['score'], x) for x in mylist]
mylist_annotated.sort()
print( [x for key, x in mylist_annotated] )
Will give:
TypeError: '<' not supported between instances of 'dict' and 'dict'
Fortunately I had an easy solution – my data had a unique key in that was sortable, so I could put that as the second key:
mylist = [{'score':50,'name':'bob','unique_id':1},{'score':50,'name':'linda','unique_id':2}]
mylist_annotated = [(x['score'], x['unique_id'], x) for x in mylist]
mylist_annotated.sort()
print( [x for key, unique, x in mylist_annotated] )
I guess if your data doesn’t have a naturally unique value in, you can insert one before trying to sort? A uuid maybe?
EDIT: As suggested in comment (Thanks!), you can also use operator.itemgetter:
import operator
mylist = [{'score':50,'name':'bob'},{'score':50,'name':'linda'}]
mylist_annotated = [(x['score'], x) for x in mylist]
mylist_annotated.sort(key=operator.itemgetter(0))
print( [x for key, x in mylist_annotated] )
An "async lambda" can be emulated by combining a lambda with an async generator:1
key=lambda x: (await somefunction(x) for _ in '_').__anext__()
It is possible to move the ( ).__anext__() to a helper, which likely makes the pattern clearer as well:
def head(async_iterator): return async_iterator.__anext__()
key=lambda x: head(await somefunction(x) for _ in '_')
Note that the sort method/function in the standard library are not async. One needs an async version, such as asyncstdlib.sorted (disclaimer: I maintain this library):
import asyncstdlib as a
mylist = await a.sorted(mylist, key=lambda x: head(await somefunction(x) for _ in '_'))
Understanding the lambda ...: (...).__anext__() pattern
An "async lambda" would be an anonymous asynchronous function, or in other words an anonymous function evaluating to an awaitable. This is in parallel to how async def defines a named function evaluating to an awaitable.
The task can be split into two parts: An anonymous function expression and a nested awaitable expression.
-
An anonymous function expression is exactly what a lambda ...: ... is.
-
An awaitable expression is only allowed inside a coroutine function; however:
- An (asynchronous) generator expression implicitly creates a (coroutine) function. As an async generator only needs async to run, it can be defined in a sync function (since Python 3.7).
- An asynchronous iterable can be used as an awaitable via its
__anext__ method.
These three parts are directly used in the "async lambda" pattern:
# | regular lambda for the callable and scope
# | | async generator expression for an async scope
# v v v first item as an awaitable
key=lambda x: (await somefunction(x) for _ in '_').__anext__()
The for _ in '_' in the async generator is only to have exactly one iteration. Any variant with at least one iteration will do.
1Be mindful whether an "async lambda" is actually needed in the first place, since async functions are first class just like regular functions. Just as lambda x: foo(x) is redundant and should just be foo, lambda x: (await bar(x) …) is redundant and should just be bar . The function body should do more than just call-and-await, such as 3 + await bar(x) or await bar(x) or await qux(x).
If you already defined a separate async function, you can simplify MisterMiyagi’s answer even a bit more:
mylist = await a.sorted(
mylist,
key=somefunction)
If you want to change the key after awaiting it, you can use asyncstdlib.apply:
mylist = await a.sorted(
mylist,
key=lambda x: a.apply(lambda after: 1 / after, some_function(x)))
Here is a complete example program:
import asyncio
import asyncstdlib as a
async def some_function(x):
return x
async def testme():
mylist=[2, 1, 3]
mylist = await a.sorted(
mylist,
key=lambda x: a.apply(lambda after: 1 / after, some_function(x)))
print(f'mylist is: {mylist}')
if __name__ == "__main__":
asyncio.run(testme())
await cannot be included in a lambda function.
The solutions here can be shortened to:
from asyncio import coroutine, run
my_list = [. . .]
async def some_function(x) -> coroutine:
. . .
my_list.sort(key=lambda x: await some_function(x)) # raises a SyntaxError
my_list.sort(key=lambda x: run(some_function(x)) # works
I’m trying to do something like this:
mylist.sort(key=lambda x: await somefunction(x))
But I get this error:
SyntaxError: 'await' outside async function
Which makes sense because the lambda is not async.
I tried to use async lambda x: ... but that throws a SyntaxError: invalid syntax.
Pep 492 states:
Syntax for asynchronous lambda functions could be provided, but this construct is outside of the scope of this PEP.
But I could not find out if that syntax was implemented in CPython.
Is there a way to declare an async lambda, or to use an async function for sorting a list?
You can’t. There is no async lambda, and even if there were, you coudln’t pass it in as key function to list.sort(), since a key function will be called as a synchronous function and not awaited. An easy work-around is to annotate your list yourself:
mylist_annotated = [(await some_function(x), x) for x in mylist]
mylist_annotated.sort()
mylist = [x for key, x in mylist_annotated]
Note that await expressions in list comprehensions are only supported in Python 3.6+. If you’re using 3.5, you can do the following:
mylist_annotated = []
for x in mylist:
mylist_annotated.append((await some_function(x), x))
mylist_annotated.sort()
mylist = [x for key, x in mylist_annotated]
The answer from Sven Marnach has an Edge case.
If you try and sort a list that has 2 items that produce the same search key but are different and are not directly sortable, it will crash.
mylist = [{'score':50,'name':'bob'},{'score':50,'name':'linda'}]
mylist_annotated = [(x['score'], x) for x in mylist]
mylist_annotated.sort()
print( [x for key, x in mylist_annotated] )
Will give:
TypeError: '<' not supported between instances of 'dict' and 'dict'
Fortunately I had an easy solution – my data had a unique key in that was sortable, so I could put that as the second key:
mylist = [{'score':50,'name':'bob','unique_id':1},{'score':50,'name':'linda','unique_id':2}]
mylist_annotated = [(x['score'], x['unique_id'], x) for x in mylist]
mylist_annotated.sort()
print( [x for key, unique, x in mylist_annotated] )
I guess if your data doesn’t have a naturally unique value in, you can insert one before trying to sort? A uuid maybe?
EDIT: As suggested in comment (Thanks!), you can also use operator.itemgetter:
import operator
mylist = [{'score':50,'name':'bob'},{'score':50,'name':'linda'}]
mylist_annotated = [(x['score'], x) for x in mylist]
mylist_annotated.sort(key=operator.itemgetter(0))
print( [x for key, x in mylist_annotated] )
An "async lambda" can be emulated by combining a lambda with an async generator:1
key=lambda x: (await somefunction(x) for _ in '_').__anext__()
It is possible to move the ( ).__anext__() to a helper, which likely makes the pattern clearer as well:
def head(async_iterator): return async_iterator.__anext__()
key=lambda x: head(await somefunction(x) for _ in '_')
Note that the sort method/function in the standard library are not async. One needs an async version, such as asyncstdlib.sorted (disclaimer: I maintain this library):
import asyncstdlib as a
mylist = await a.sorted(mylist, key=lambda x: head(await somefunction(x) for _ in '_'))
Understanding the lambda ...: (...).__anext__() pattern
An "async lambda" would be an anonymous asynchronous function, or in other words an anonymous function evaluating to an awaitable. This is in parallel to how async def defines a named function evaluating to an awaitable.
The task can be split into two parts: An anonymous function expression and a nested awaitable expression.
-
An anonymous function expression is exactly what a
lambda ...: ...is. -
An awaitable expression is only allowed inside a coroutine function; however:
- An (asynchronous) generator expression implicitly creates a (coroutine) function. As an async generator only needs async to run, it can be defined in a sync function (since Python 3.7).
- An asynchronous iterable can be used as an awaitable via its
__anext__method.
These three parts are directly used in the "async lambda" pattern:
# | regular lambda for the callable and scope
# | | async generator expression for an async scope
# v v v first item as an awaitable
key=lambda x: (await somefunction(x) for _ in '_').__anext__()
The for _ in '_' in the async generator is only to have exactly one iteration. Any variant with at least one iteration will do.
1Be mindful whether an "async lambda" is actually needed in the first place, since async functions are first class just like regular functions. Just as lambda x: foo(x) is redundant and should just be foo, lambda x: (await bar(x) …) is redundant and should just be bar . The function body should do more than just call-and-await, such as 3 + await bar(x) or await bar(x) or await qux(x).
If you already defined a separate async function, you can simplify MisterMiyagi’s answer even a bit more:
mylist = await a.sorted(
mylist,
key=somefunction)
If you want to change the key after awaiting it, you can use asyncstdlib.apply:
mylist = await a.sorted(
mylist,
key=lambda x: a.apply(lambda after: 1 / after, some_function(x)))
Here is a complete example program:
import asyncio
import asyncstdlib as a
async def some_function(x):
return x
async def testme():
mylist=[2, 1, 3]
mylist = await a.sorted(
mylist,
key=lambda x: a.apply(lambda after: 1 / after, some_function(x)))
print(f'mylist is: {mylist}')
if __name__ == "__main__":
asyncio.run(testme())
await cannot be included in a lambda function.
The solutions here can be shortened to:
from asyncio import coroutine, run
my_list = [. . .]
async def some_function(x) -> coroutine:
. . .
my_list.sort(key=lambda x: await some_function(x)) # raises a SyntaxError
my_list.sort(key=lambda x: run(some_function(x)) # works