How can I access variables from the caller, even if it isn't an enclosing scope (i.e., implement dynamic scoping)?
Question:
Consider this example:
def outer():
s_outer = "outern"
def inner():
s_inner = "innern"
do_something()
inner()
I want the code in do_something
to be able to access the variables of the calling functions further up the call stack, in this case s_outer
and s_inner
. More generally, I want to call it from various other functions, but always execute it in their respective context and access their respective scopes (implement dynamic scoping).
I know that in Python 3.x, the nonlocal
keyword allows access to s_outer
from within inner
. Unfortunately, that only helps with do_something
if it’s defined within inner
. Otherwise, inner
isn’t a lexically enclosing scope (similarly, neither is outer
, unless do_something
is defined within outer
).
I figured out how to inspect stack frames with the standard library inspect
, and made a small accessor that I can call from within do_something()
like this:
def reach(name):
for f in inspect.stack():
if name in f[0].f_locals:
return f[0].f_locals[name]
return None
and then
def do_something():
print( reach("s_outer"), reach("s_inner") )
works just fine.
Can reach
be implemented more simply? How else can I solve the problem?
Answers:
There is no and, in my opinion, should be no elegant way of implementing reach
since that introduces a new non-standard indirection which is really hard to comprehend, debug, test and maintain. As the Python mantra (try import this
) says:
Explicit is better than implicit.
So, just pass the arguments. You-from-the-future will be really grateful to you-from-today.
What I ended up doing was
scope = locals()
and make scope
accessible from do_something
. That way I don’t have to reach, but I can still access the dictionary of local variables of the caller. This is quite similar to building a dictionary myself and passing it on.
Is there a better way to solve this problem? (Other than wrapping the respective data into dicts and pass these dicts explicitly to do_something())
Passing the dicts explicitly is a better way.
What you’re proposing sounds very unconventional. When code increases in size, you have to break down the code into a modular architecture, with clean APIs between modules. It also has to be something that is easy to comprehend, easy to explain, and easy to hand over to another programmer to modify/improve/debug it. What you’re proposing sounds like it is not a clean API, unconventional, with a non-obvious data flow. I suspect it would probably make many programmers grumpy when they saw it. 🙂
Another option would be to make the functions members of a class, with the data being in the class instance. That could work well if your problem can be modelled as several functions operating on the data object.
We can get naughtier.
This is an answer to the "Is there a more elegant/shortened way to implement the reach()
function?" half of the question.
-
We can give better syntax for the user: instead of reach("foo")
, outer.foo
.
This is nicer to type, and the language itself immediately tells you if you used a name that can’t be a valid variable (attribute names and variable names have the same constraints).
-
We can raise an error, to properly distinguish "this doesn’t exist" from "this was set to None
".
If we actually want to smudge those cases together, we can getattr
with the default parameter, or try
–except AttributeError
.
-
We can optimize: no need to pessimistically build a list big enough for all the frames at once.
In most cases we probably won’t need to go all the way to the root of the call stack.
-
Just because we’re inappropriately reaching up stack frames, violating one of the most important rules of programming to not have things far away invisibly effecting behavior, doesn’t mean we can’t be civilized.
If someone is trying to use this Serious API for Real Work on a Python without stack frame inspection support, we should helpfully let them know.
import inspect
class OuterScopeGetter(object):
def __getattribute__(self, name):
frame = inspect.currentframe()
if frame is None:
raise RuntimeError('cannot inspect stack frames')
sentinel = object()
frame = frame.f_back
while frame is not None:
value = frame.f_locals.get(name, sentinel)
if value is not sentinel:
return value
frame = frame.f_back
raise AttributeError(repr(name) + ' not found in any outer scope')
outer = OuterScopeGetter()
Excellent. Now we can just do:
>>> def f():
... return outer.x
...
>>> f()
Traceback (most recent call last):
...
AttributeError: 'x' not found in any outer scope
>>>
>>> x = 1
>>> f()
1
>>> x = 2
>>> f()
2
>>>
>>> def do_something():
... print(outer.y)
... print(outer.z)
...
>>> def g():
... y = 3
... def h():
... z = 4
... do_something()
... h()
...
>>> g()
3
4
Perversion elegantly achieved.
(P.S. This is a simplified read-only version of a more complete implementation in my dynamicscope
library.)
Consider this example:
def outer():
s_outer = "outern"
def inner():
s_inner = "innern"
do_something()
inner()
I want the code in do_something
to be able to access the variables of the calling functions further up the call stack, in this case s_outer
and s_inner
. More generally, I want to call it from various other functions, but always execute it in their respective context and access their respective scopes (implement dynamic scoping).
I know that in Python 3.x, the nonlocal
keyword allows access to s_outer
from within inner
. Unfortunately, that only helps with do_something
if it’s defined within inner
. Otherwise, inner
isn’t a lexically enclosing scope (similarly, neither is outer
, unless do_something
is defined within outer
).
I figured out how to inspect stack frames with the standard library inspect
, and made a small accessor that I can call from within do_something()
like this:
def reach(name):
for f in inspect.stack():
if name in f[0].f_locals:
return f[0].f_locals[name]
return None
and then
def do_something():
print( reach("s_outer"), reach("s_inner") )
works just fine.
Can reach
be implemented more simply? How else can I solve the problem?
There is no and, in my opinion, should be no elegant way of implementing reach
since that introduces a new non-standard indirection which is really hard to comprehend, debug, test and maintain. As the Python mantra (try import this
) says:
Explicit is better than implicit.
So, just pass the arguments. You-from-the-future will be really grateful to you-from-today.
What I ended up doing was
scope = locals()
and make scope
accessible from do_something
. That way I don’t have to reach, but I can still access the dictionary of local variables of the caller. This is quite similar to building a dictionary myself and passing it on.
Is there a better way to solve this problem? (Other than wrapping the respective data into dicts and pass these dicts explicitly to do_something())
Passing the dicts explicitly is a better way.
What you’re proposing sounds very unconventional. When code increases in size, you have to break down the code into a modular architecture, with clean APIs between modules. It also has to be something that is easy to comprehend, easy to explain, and easy to hand over to another programmer to modify/improve/debug it. What you’re proposing sounds like it is not a clean API, unconventional, with a non-obvious data flow. I suspect it would probably make many programmers grumpy when they saw it. 🙂
Another option would be to make the functions members of a class, with the data being in the class instance. That could work well if your problem can be modelled as several functions operating on the data object.
We can get naughtier.
This is an answer to the "Is there a more elegant/shortened way to implement the reach()
function?" half of the question.
-
We can give better syntax for the user: instead of
reach("foo")
,outer.foo
.This is nicer to type, and the language itself immediately tells you if you used a name that can’t be a valid variable (attribute names and variable names have the same constraints).
-
We can raise an error, to properly distinguish "this doesn’t exist" from "this was set to
None
".If we actually want to smudge those cases together, we can
getattr
with the default parameter, ortry
–except AttributeError
. -
We can optimize: no need to pessimistically build a list big enough for all the frames at once.
In most cases we probably won’t need to go all the way to the root of the call stack.
-
Just because we’re inappropriately reaching up stack frames, violating one of the most important rules of programming to not have things far away invisibly effecting behavior, doesn’t mean we can’t be civilized.
If someone is trying to use this Serious API for Real Work on a Python without stack frame inspection support, we should helpfully let them know.
import inspect
class OuterScopeGetter(object):
def __getattribute__(self, name):
frame = inspect.currentframe()
if frame is None:
raise RuntimeError('cannot inspect stack frames')
sentinel = object()
frame = frame.f_back
while frame is not None:
value = frame.f_locals.get(name, sentinel)
if value is not sentinel:
return value
frame = frame.f_back
raise AttributeError(repr(name) + ' not found in any outer scope')
outer = OuterScopeGetter()
Excellent. Now we can just do:
>>> def f():
... return outer.x
...
>>> f()
Traceback (most recent call last):
...
AttributeError: 'x' not found in any outer scope
>>>
>>> x = 1
>>> f()
1
>>> x = 2
>>> f()
2
>>>
>>> def do_something():
... print(outer.y)
... print(outer.z)
...
>>> def g():
... y = 3
... def h():
... z = 4
... do_something()
... h()
...
>>> g()
3
4
Perversion elegantly achieved.
(P.S. This is a simplified read-only version of a more complete implementation in my dynamicscope
library.)