How to define free-variable in python?
Question:
The local/global/free variable definitions from python doc:
If a name is bound in a block, it is a local variable of that block, unless declared as nonlocal. If a name is bound at the module level, it is a global variable. (The variables of the module code block are local and global.) If a variable is used in a code block but not defined there, it is a free variable.
Code 1:
>>> x = 0
>>> def foo():
... print(x)
... print(locals())
...
>>> foo()
0
{}
Code 2:
>>> def bar():
... x = 1
... def foo():
... print(x)
... print(locals())
... foo()
...
>>> bar()
1
{'x':1}
Free variables are returned by locals() when it is called in function blocks, but not in class blocks.
In Code 1, x is a global variable, and it’s used but not defined in foo().
However it’s not a free variable, because it’s not returned by locals().
I think it’s not what the doc said. Is there a technical definition for free variable?
Answers:
Definition of a free variable: Used, but neither global nor bound.
For example:
x is not free in Code 1, because it’s a global variable.x is not free in bar() in Code 2, because it’s a bound variable.x is free in foo().
Python makes this distinction because of closures. A free variable is not defined in the current environment, i. e. collection of local variables, and is also not a global variable! Therefore it must be defined elsewhere. And this is the concept of closures. In Code 2, foo() closes on x defined in bar(). Python uses lexical scope. This means, the interpreter is able to determine the scope by just looking at the code.
For example: x is known as a variable in foo(), because foo() is enclosed by bar(), and x is bound in bar().
Global scope is treated specially by Python. It would be possible to view the global scope as an outermost scope, but this is not done because of performance (I think). Therefore it is not possible that x is both free and global.
Exemption
Life is not so simple. There exist free global variables. Python docs (Execution model) says:
The global statement has the same scope as a name binding operation in the same block. If the nearest enclosing scope for a free variable contains a global statement, the free variable is treated as a global.
>>> x = 42
>>> def foo():
... global x
... def baz():
... print(x)
... print(locals())
... baz()
...
>>> foo()
42
{}
I didn’t know that myself. We are all here to learn.
From what I have understood the documentation is indeed a bit ambiguous on free variables. There are free global variables which are treated as plain globals and lexically bound free variables. Eli Bendersky sum’s it up nicely in a blog post on symbol tables:
Unfortunately, there’s a shorthand in the core of Python that may initially confuse readers as to exactly what constitutes a "free" variable. Fortunately, it’s a very slight confusion that’s easy to put in order. The execution model reference says:
If a variable is used in a code block but not defined there, it is a free variable.
This is consistent with the formal definition. In the source, however, "free" is actually used as a shorthand for "lexically bound free variable" (i.e. variables for which a binding has been found in an enclosing scope), with "global" being used to refer to all remaining free variables. So when reading the CPython source code it is important to remember that the full set of free variables includes both the variables tagged specifically as "free", as well as those tagged as "global".
Thus, to avoid a confusion I say "lexically bound" when I want to refer to the variables actually treated in CPython as free.
(emphasis mine)
The reason why this shorthand was used is probably because when you have a global free variable there’s really no change whatsoever in the bytecode emitted. If a global variable is ‘free’ or if it isn’t doesn’t change the fact that the look-up for that name will use LOAD_GLOBAL in both cases. So global free variables aren’t all that special.
On the other hand, lexically bound variables are treated specially and are enclosed in cell objects, the objects are the storage space for lexically bound free variables and are located in the __closure__ attribute for a given function. A special LOAD_DEREF instruction is created for these that examines the cells present for the free variables. The description for the LOAD_DEREF instruction is:
LOAD_DEREF(i)
Loads the cell contained in slot i of the cell and free variable storage
So in Python free variables only make a difference as a concept in situations where a definition for an object that has state is lexically (i.e statically) nested in another definition for an object that has state.
Variables are nothing but reserved memory locations to store values. This means that when you create a variable you reserve some space in memory.
Based on the data type of a variable, the interpreter allocates memory and decides what can be stored in the reserved memory. Therefore, by assigning different data types to variables, you can store integers, decimals or characters in these variables.
Assigning Values to Variables
Python variables do not need explicit declaration to reserve memory space. The declaration happens automatically when you assign a value to a variable. The equal sign (=) is used to assign values to variables.
There is no explicit keyword to declare a free variable in Python. Based on the definition of a function and the statements inside and surrounding it, Python will classify variables into bound, cell and free variables.
The following example illustrates this concept using the function’s code object which encapsulates the variables mentioned in the previous paragraph.
def func(arg1, arg2=2):
def inner_func(arg3=arg2):
return arg1, arg3
arg1, arg2 = None
return inner_func
For ‘func‘:
• arg1 and arg2 are bound variables
• arg1 is a cell variable since it is a free variable inside of ‘inner_func‘
• There are no free variables.
func.__code__.co_varnames
(‘arg1’, ‘arg2’, ‘inner_func’)
func.__code__.co_cellvars
(‘arg1’,)
func.__code__.co_freevars
()
For ‘inner_func‘:
• arg3 is a bound variable
• arg1 is a free variable
• There are no cell variables
inner_func.__code__.co_varnames
(‘arg3’,)
inner_func.__code__.co_freevars
(‘arg1’)
inner_func.__code__.co_cellvars
()
Just so that novices don’t get mislead, the comment above by Nikhil starting with “Variables are nothing but reserved memory locations to store values.” is completely wrong for Python.
In Python there are “names” and there are “values”. Values have types, not names. Memory space is reserved for values, not for names. E.g., we can have x = 1 followed later in the code by x = “a string” followed later by x = [3, 9.1]. Over the course of these assignments the name x first points to an integer, then to a string, and finally to a list. When an assignment is done, the name on the left side of the assignment is made to point to the value on the right hand side of the assignment. Values can be un-changeable (immutable) or changeable (mutable). Integers, strings, tuples, etc are immutable; lists, etc are mutable. Since integers are immutable, when there are two statements like this:
x = 4
x = x +1
the second statement makes x point to a new value 5, it is not changing the value in the memory location pointed at by x from 4 to 5!
This is a very different model from that of say the C language!
The local/global/free variable definitions from python doc:
If a name is bound in a block, it is a local variable of that block, unless declared as nonlocal. If a name is bound at the module level, it is a global variable. (The variables of the module code block are local and global.) If a variable is used in a code block but not defined there, it is a free variable.
Code 1:
>>> x = 0
>>> def foo():
... print(x)
... print(locals())
...
>>> foo()
0
{}
Code 2:
>>> def bar():
... x = 1
... def foo():
... print(x)
... print(locals())
... foo()
...
>>> bar()
1
{'x':1}
Free variables are returned by locals() when it is called in function blocks, but not in class blocks.
In Code 1, x is a global variable, and it’s used but not defined in foo().
However it’s not a free variable, because it’s not returned by locals().
I think it’s not what the doc said. Is there a technical definition for free variable?
Definition of a free variable: Used, but neither global nor bound.
For example:
xis not free in Code 1, because it’s a global variable.xis not free inbar()in Code 2, because it’s a bound variable.xis free infoo().
Python makes this distinction because of closures. A free variable is not defined in the current environment, i. e. collection of local variables, and is also not a global variable! Therefore it must be defined elsewhere. And this is the concept of closures. In Code 2, foo() closes on x defined in bar(). Python uses lexical scope. This means, the interpreter is able to determine the scope by just looking at the code.
For example: x is known as a variable in foo(), because foo() is enclosed by bar(), and x is bound in bar().
Global scope is treated specially by Python. It would be possible to view the global scope as an outermost scope, but this is not done because of performance (I think). Therefore it is not possible that x is both free and global.
Exemption
Life is not so simple. There exist free global variables. Python docs (Execution model) says:
The global statement has the same scope as a name binding operation in the same block. If the nearest enclosing scope for a free variable contains a global statement, the free variable is treated as a global.
>>> x = 42
>>> def foo():
... global x
... def baz():
... print(x)
... print(locals())
... baz()
...
>>> foo()
42
{}
I didn’t know that myself. We are all here to learn.
From what I have understood the documentation is indeed a bit ambiguous on free variables. There are free global variables which are treated as plain globals and lexically bound free variables. Eli Bendersky sum’s it up nicely in a blog post on symbol tables:
Unfortunately, there’s a shorthand in the core of Python that may initially confuse readers as to exactly what constitutes a "free" variable. Fortunately, it’s a very slight confusion that’s easy to put in order. The execution model reference says:
If a variable is used in a code block but not defined there, it is a free variable.
This is consistent with the formal definition. In the source, however, "free" is actually used as a shorthand for "lexically bound free variable" (i.e. variables for which a binding has been found in an enclosing scope), with "global" being used to refer to all remaining free variables. So when reading the CPython source code it is important to remember that the full set of free variables includes both the variables tagged specifically as "free", as well as those tagged as "global".
Thus, to avoid a confusion I say "lexically bound" when I want to refer to the variables actually treated in CPython as free.
(emphasis mine)
The reason why this shorthand was used is probably because when you have a global free variable there’s really no change whatsoever in the bytecode emitted. If a global variable is ‘free’ or if it isn’t doesn’t change the fact that the look-up for that name will use LOAD_GLOBAL in both cases. So global free variables aren’t all that special.
On the other hand, lexically bound variables are treated specially and are enclosed in cell objects, the objects are the storage space for lexically bound free variables and are located in the __closure__ attribute for a given function. A special LOAD_DEREF instruction is created for these that examines the cells present for the free variables. The description for the LOAD_DEREF instruction is:
LOAD_DEREF(i)Loads the cell contained in slot i of the cell and free variable storage
So in Python free variables only make a difference as a concept in situations where a definition for an object that has state is lexically (i.e statically) nested in another definition for an object that has state.
Variables are nothing but reserved memory locations to store values. This means that when you create a variable you reserve some space in memory.
Based on the data type of a variable, the interpreter allocates memory and decides what can be stored in the reserved memory. Therefore, by assigning different data types to variables, you can store integers, decimals or characters in these variables.
Assigning Values to Variables
Python variables do not need explicit declaration to reserve memory space. The declaration happens automatically when you assign a value to a variable. The equal sign (=) is used to assign values to variables.
There is no explicit keyword to declare a free variable in Python. Based on the definition of a function and the statements inside and surrounding it, Python will classify variables into bound, cell and free variables.
The following example illustrates this concept using the function’s code object which encapsulates the variables mentioned in the previous paragraph.
def func(arg1, arg2=2):
def inner_func(arg3=arg2):
return arg1, arg3
arg1, arg2 = None
return inner_func
For ‘func‘:
• arg1 and arg2 are bound variables
• arg1 is a cell variable since it is a free variable inside of ‘inner_func‘
• There are no free variables.
func.__code__.co_varnames
(‘arg1’, ‘arg2’, ‘inner_func’)
func.__code__.co_cellvars
(‘arg1’,)
func.__code__.co_freevars
()
For ‘inner_func‘:
• arg3 is a bound variable
• arg1 is a free variable
• There are no cell variables
inner_func.__code__.co_varnames
(‘arg3’,)
inner_func.__code__.co_freevars
(‘arg1’)
inner_func.__code__.co_cellvars
()
Just so that novices don’t get mislead, the comment above by Nikhil starting with “Variables are nothing but reserved memory locations to store values.” is completely wrong for Python.
In Python there are “names” and there are “values”. Values have types, not names. Memory space is reserved for values, not for names. E.g., we can have x = 1 followed later in the code by x = “a string” followed later by x = [3, 9.1]. Over the course of these assignments the name x first points to an integer, then to a string, and finally to a list. When an assignment is done, the name on the left side of the assignment is made to point to the value on the right hand side of the assignment. Values can be un-changeable (immutable) or changeable (mutable). Integers, strings, tuples, etc are immutable; lists, etc are mutable. Since integers are immutable, when there are two statements like this:
x = 4
x = x +1
the second statement makes x point to a new value 5, it is not changing the value in the memory location pointed at by x from 4 to 5!
This is a very different model from that of say the C language!