In Python, why can a lambda expression refer to the variable being defined but not a list?
Question:
This is more a curiosity than anything, but I just noticed the following. If I am defining a self-referential lambda, I can do it easily:
>>> f = lambda: f
>>> f() is f
True
But if I am defining a self-referential list, I have to do it in more than one statement:
>>> a = [a]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined
>>> a = []
>>> a.append(a)
>>> a[0] is a
True
>>> a
[[...]]
I also noticed that this is not limited to lists but seems like any other expression other than a lambda can not reference the variable left of the assignment. For example, if you have a cyclic linked-list with one node, you can’t simply go:
>>> class Node(object):
... def __init__(self, next_node):
... self.next = next_node
...
>>> n = Node(n)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'n' is not defined
Instead, you have to do it in two statements:
>>> n = Node(None)
>>> n.next = n
>>> n is n.next
True
Does anyone know what the philosophy behind this difference is? I understand that a recursive lambda are used much more frequently, and hence supporting self-reference is important for lambdas, but why not allow it for any assignment?
EDIT: The answers below clarify this quite nicely. The reason is that variables in lambdas in Python are evaluated each time the lambda is called, not when it’s defined. In this sense they are exactly like functions defined using def. I wrote the following bit of code to experiment with how this works, both with lambdas and def functions in case it might help clarify it for anyone.
>>> f = lambda: f
>>> f() is f
True
>>> g = f
>>> f = "something else"
>>> g()
'something else'
>>> f = "hello"
>>> g()
'hello'
>>> f = g
>>> g() is f
True
>>> def f():
... print(f)
...
>>> f()
<function f at 0x10d125560>
>>> g = f
>>> g()
<function f at 0x10d125560>
>>> f = "test"
>>> g()
test
>>> f = "something else"
>>> g()
something else
Answers:
The expression inside a lambda is evaluated when the function is called, not when it is defined.
In other words, Python will not evaluate the f inside your lambda until you call it. And, by then, f is already defined in the current scope (it is the lambda itself). Hence, no NameError is raised.
Note that this is not the case for a line like this:
a = [a]
When Python interprets this type of line (known as an assignment statement), it will evaluate the expression on the right of the = immediately. Moreover, a NameError will be raised for any name used on the right that is undefined in the current scope.
Because a lambda is a function, and the function body is not executed until the function is called.
In other words, the other way to do it is this:
def f():
return f
But you’re correct that you can’t do it in an expression because def is a statement, so it can’t be used in an expression.
There’s no special-casing required to make this happen; it’s just how it works.
A lambda expression is not any different from a normal function, really. Meaning, I can do this:
x = 1
def f():
print x + 2
f()
3
x = 2
f()
4
As you can see, inside the function, the value of x does not have a predefined value – it’s looked up when we actually run f. This includes the value of the function itself: We don’t look up what f represents until we actually run it, and by then it exists.
Doing that as a lambda doesn’t work any differently:
del x
f = lambda: x+2
f()
NameError: global name 'x' is not defined
x = 2
f()
4
works similarly. In this case, I went ahead and deleted x so it was no longer in the scope when f was defined, and running f in this case correctly shows that x doesn’t exist. But after we define x, then f works again.
This is different in the list case, because we are actually generating an object right now, and so everything on the right side has to be bound, right now. The way python works (as i understand it, and at least in practice this has been useful) is to consider that everything on the right side is deferenced & bound and then processed, and only after that’s all complete are the value(s) on the left side bound and assigned.
Since the same value is on the right side and the left, when python tries to bind the name on the right side, it doesn’t exist yet.
We can see when we disassemble the lambda function (this is identical output in Python 2.6 and 3.3)
>>> import dis
>>> f = lambda: f
>>> dis.dis(f)
1 0 LOAD_GLOBAL 0 (f)
3 RETURN_VALUE
We demonstrate that we do not need to load f until it is called, whereupon it is already defined globally, and therefore stored, so this works:
>>> f is f()
True
But when we do:
>>> a = [a]
We have an error (if a is previously undefined), and if we disassemble Python’s implementation of this.
>>> def foo():
... a = [a]
...
>>> dis.dis(foo)
2 0 LOAD_FAST 0 (a)
3 BUILD_LIST 1
6 STORE_FAST 0 (a)
9 LOAD_CONST 0 (None)
12 RETURN_VALUE
we see that we attempt to load a before we have stored it.
This is more a curiosity than anything, but I just noticed the following. If I am defining a self-referential lambda, I can do it easily:
>>> f = lambda: f
>>> f() is f
True
But if I am defining a self-referential list, I have to do it in more than one statement:
>>> a = [a]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined
>>> a = []
>>> a.append(a)
>>> a[0] is a
True
>>> a
[[...]]
I also noticed that this is not limited to lists but seems like any other expression other than a lambda can not reference the variable left of the assignment. For example, if you have a cyclic linked-list with one node, you can’t simply go:
>>> class Node(object):
... def __init__(self, next_node):
... self.next = next_node
...
>>> n = Node(n)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'n' is not defined
Instead, you have to do it in two statements:
>>> n = Node(None)
>>> n.next = n
>>> n is n.next
True
Does anyone know what the philosophy behind this difference is? I understand that a recursive lambda are used much more frequently, and hence supporting self-reference is important for lambdas, but why not allow it for any assignment?
EDIT: The answers below clarify this quite nicely. The reason is that variables in lambdas in Python are evaluated each time the lambda is called, not when it’s defined. In this sense they are exactly like functions defined using def. I wrote the following bit of code to experiment with how this works, both with lambdas and def functions in case it might help clarify it for anyone.
>>> f = lambda: f
>>> f() is f
True
>>> g = f
>>> f = "something else"
>>> g()
'something else'
>>> f = "hello"
>>> g()
'hello'
>>> f = g
>>> g() is f
True
>>> def f():
... print(f)
...
>>> f()
<function f at 0x10d125560>
>>> g = f
>>> g()
<function f at 0x10d125560>
>>> f = "test"
>>> g()
test
>>> f = "something else"
>>> g()
something else
The expression inside a lambda is evaluated when the function is called, not when it is defined.
In other words, Python will not evaluate the f inside your lambda until you call it. And, by then, f is already defined in the current scope (it is the lambda itself). Hence, no NameError is raised.
Note that this is not the case for a line like this:
a = [a]
When Python interprets this type of line (known as an assignment statement), it will evaluate the expression on the right of the = immediately. Moreover, a NameError will be raised for any name used on the right that is undefined in the current scope.
Because a lambda is a function, and the function body is not executed until the function is called.
In other words, the other way to do it is this:
def f():
return f
But you’re correct that you can’t do it in an expression because def is a statement, so it can’t be used in an expression.
There’s no special-casing required to make this happen; it’s just how it works.
A lambda expression is not any different from a normal function, really. Meaning, I can do this:
x = 1
def f():
print x + 2
f()
3
x = 2
f()
4
As you can see, inside the function, the value of x does not have a predefined value – it’s looked up when we actually run f. This includes the value of the function itself: We don’t look up what f represents until we actually run it, and by then it exists.
Doing that as a lambda doesn’t work any differently:
del x
f = lambda: x+2
f()
NameError: global name 'x' is not defined
x = 2
f()
4
works similarly. In this case, I went ahead and deleted x so it was no longer in the scope when f was defined, and running f in this case correctly shows that x doesn’t exist. But after we define x, then f works again.
This is different in the list case, because we are actually generating an object right now, and so everything on the right side has to be bound, right now. The way python works (as i understand it, and at least in practice this has been useful) is to consider that everything on the right side is deferenced & bound and then processed, and only after that’s all complete are the value(s) on the left side bound and assigned.
Since the same value is on the right side and the left, when python tries to bind the name on the right side, it doesn’t exist yet.
We can see when we disassemble the lambda function (this is identical output in Python 2.6 and 3.3)
>>> import dis
>>> f = lambda: f
>>> dis.dis(f)
1 0 LOAD_GLOBAL 0 (f)
3 RETURN_VALUE
We demonstrate that we do not need to load f until it is called, whereupon it is already defined globally, and therefore stored, so this works:
>>> f is f()
True
But when we do:
>>> a = [a]
We have an error (if a is previously undefined), and if we disassemble Python’s implementation of this.
>>> def foo():
... a = [a]
...
>>> dis.dis(foo)
2 0 LOAD_FAST 0 (a)
3 BUILD_LIST 1
6 STORE_FAST 0 (a)
9 LOAD_CONST 0 (None)
12 RETURN_VALUE
we see that we attempt to load a before we have stored it.