Max recursion is not exactly what sys.getrecursionlimit() claims. How come?
Question:
I’ve made a small function that will actually measure the max recursion limit:
def f(x):
r = x
try:
r = f(x+1)
except Exception as e:
print(e)
finally:
return r
To know what to expect I’ve checked:
In [28]: import sys
In [29]: sys.getrecursionlimit()
Out[29]: 1000
However
In [30]: f(0)
maximum recursion depth exceeded
Out[30]: 970
The number is not fixed, always around ~970, and slightly changes between different instances of python (e.g. from within spyder to system cmd prompt).
Please note that I’m using ipython on python3.
What’s going on? Why is the actual limit I’m getting lower than the sys.getrecursionlimit() value?
Answers:
The recursion limit is not the limit on recursion but the maximum depth of the python interpreter stack.There is something on the stack before your function gets executed. Spyder executes some python stuff before it calls your script, as do other interpreters like ipython.
You can inspect the stack via methods in the inspect module.
In CPython for me:
>>>print(len(inspect.stack()))
1
In Ipython for me:
>>>print(len(inspect.stack()))
10
As knbk pointed out in the comments as soon as you hit the stack limit a RecursionError is thrown and the interpreter raises the stack limit a bit to give you a possibility to handle the error gracefully. If you also exhaust that limit python will crash.
This limit is for stack, not for the function you define. There are other internal things which might push something to stack.
And of course it could depend on env in which it was executed. Some can pollute stack more, some less.
I believe the confusion originates from the difference between the stack size you see when the error occurs and the limit. The thing is that the last call, that caused the crash, likely accounts for more than 1 frame on the stack, because it itself makes some function calls. And by the time you catch the exception, the call and its internal calls will have been removed from the stack. You can actually see them in the traceback. Let’s look at this one.
In [1]: import inspect
In [2]: import sys
In [3]: sys.setrecursionlimit(50) # I'm setting this to 50 to make the traceback shorter.
In [4]: stack_log = []
In [5]: def recur():
stack_log.append(len(inspect.stack()))
recur()
...:
In [6]: recur()
We get the traceback (note: there is no need to read it now, so just move forward to the next section).
---------------------------------------------------------------------------
RecursionError Traceback (most recent call last)
<ipython-input-6-45136123341b> in <module>()
----> 1 recur()
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
----> 2 stack_log.append(len(inspect.stack()))
3 recur()
4
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in stack(context)
1462 def stack(context=1):
1463 """Return a list of records for the stack above the caller's frame."""
-> 1464 return getouterframes(sys._getframe(1), context)
1465
1466 def trace(context=1):
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in getouterframes(frame, context)
1439 framelist = []
1440 while frame:
-> 1441 frameinfo = (frame,) + getframeinfo(frame, context)
1442 framelist.append(FrameInfo(*frameinfo))
1443 frame = frame.f_back
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in getframeinfo(frame, context)
1412 start = lineno - 1 - context//2
1413 try:
-> 1414 lines, lnum = findsource(frame)
1415 except OSError:
1416 lines = index = None
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in findsource(object)
742 is raised if the source code cannot be retrieved."""
743
--> 744 file = getsourcefile(object)
745 if file:
746 # Invalidate cache if needed.
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in getsourcefile(object)
670 return filename
671 # only return a non-existent filename if the module has a PEP 302 loader
--> 672 if getattr(getmodule(object, filename), '__loader__', None) is not None:
673 return filename
674 # or it is in the linecache
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in getmodule(object, _filename)
699 # Try the cache again with the absolute file name
700 try:
--> 701 file = getabsfile(object, _filename)
702 except TypeError:
703 return None
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in getabsfile(object, _filename)
683 if _filename is None:
684 _filename = getsourcefile(object) or getfile(object)
--> 685 return os.path.normcase(os.path.abspath(_filename))
686
687 modulesbyfile = {}
/Users/ilia/.venvs/py3/bin/../lib/python3.5/posixpath.py in abspath(path)
355 def abspath(path):
356 """Return an absolute path."""
--> 357 if not isabs(path):
358 if isinstance(path, bytes):
359 cwd = os.getcwdb()
/Users/ilia/.venvs/py3/bin/../lib/python3.5/posixpath.py in isabs(s)
61 def isabs(s):
62 """Test whether a path is absolute"""
---> 63 sep = _get_sep(s)
64 return s.startswith(sep)
65
RecursionError: maximum recursion depth exceeded
What’s with the stack log?
In [7]: stack_log[-1]
Out[7]: 39
Okay, we have 11 missing frames. Now, scroll down the traceback to the last recur call, i.e.
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
----> 2 stack_log.append(len(inspect.stack()))
3 recur()
4
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in stack(context)
1462 def stack(context=1):
1463 """Return a list of records for the stack above the caller's frame."""
-> 1464 return getouterframes(sys._getframe(1), context)
1465
1466 def trace(context=1):
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in getouterframes(frame, context)
1439 framelist = []
1440 while frame:
-> 1441 frameinfo = (frame,) + getframeinfo(frame, context)
1442 framelist.append(FrameInfo(*frameinfo))
1443 frame = frame.f_back
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in getframeinfo(frame, context)
1412 start = lineno - 1 - context//2
1413 try:
-> 1414 lines, lnum = findsource(frame)
1415 except OSError:
1416 lines = index = None
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in findsource(object)
742 is raised if the source code cannot be retrieved."""
743
--> 744 file = getsourcefile(object)
745 if file:
746 # Invalidate cache if needed.
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in getsourcefile(object)
670 return filename
671 # only return a non-existent filename if the module has a PEP 302 loader
--> 672 if getattr(getmodule(object, filename), '__loader__', None) is not None:
673 return filename
674 # or it is in the linecache
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in getmodule(object, _filename)
699 # Try the cache again with the absolute file name
700 try:
--> 701 file = getabsfile(object, _filename)
702 except TypeError:
703 return None
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in getabsfile(object, _filename)
683 if _filename is None:
684 _filename = getsourcefile(object) or getfile(object)
--> 685 return os.path.normcase(os.path.abspath(_filename))
686
687 modulesbyfile = {}
/Users/ilia/.venvs/py3/bin/../lib/python3.5/posixpath.py in abspath(path)
355 def abspath(path):
356 """Return an absolute path."""
--> 357 if not isabs(path):
358 if isinstance(path, bytes):
359 cwd = os.getcwdb()
/Users/ilia/.venvs/py3/bin/../lib/python3.5/posixpath.py in isabs(s)
61 def isabs(s):
62 """Test whether a path is absolute"""
---> 63 sep = _get_sep(s)
64 return s.startswith(sep)
65
RecursionError: maximum recursion depth exceeded
And here you are, there are exactly 11 function calls (the arrows on the left), that is 11 frames on the stack that were removed when the exception was raised.
I’ve made a small function that will actually measure the max recursion limit:
def f(x):
r = x
try:
r = f(x+1)
except Exception as e:
print(e)
finally:
return r
To know what to expect I’ve checked:
In [28]: import sys
In [29]: sys.getrecursionlimit()
Out[29]: 1000
However
In [30]: f(0)
maximum recursion depth exceeded
Out[30]: 970
The number is not fixed, always around ~970, and slightly changes between different instances of python (e.g. from within spyder to system cmd prompt).
Please note that I’m using ipython on python3.
What’s going on? Why is the actual limit I’m getting lower than the sys.getrecursionlimit() value?
The recursion limit is not the limit on recursion but the maximum depth of the python interpreter stack.There is something on the stack before your function gets executed. Spyder executes some python stuff before it calls your script, as do other interpreters like ipython.
You can inspect the stack via methods in the inspect module.
In CPython for me:
>>>print(len(inspect.stack()))
1
In Ipython for me:
>>>print(len(inspect.stack()))
10
As knbk pointed out in the comments as soon as you hit the stack limit a RecursionError is thrown and the interpreter raises the stack limit a bit to give you a possibility to handle the error gracefully. If you also exhaust that limit python will crash.
This limit is for stack, not for the function you define. There are other internal things which might push something to stack.
And of course it could depend on env in which it was executed. Some can pollute stack more, some less.
I believe the confusion originates from the difference between the stack size you see when the error occurs and the limit. The thing is that the last call, that caused the crash, likely accounts for more than 1 frame on the stack, because it itself makes some function calls. And by the time you catch the exception, the call and its internal calls will have been removed from the stack. You can actually see them in the traceback. Let’s look at this one.
In [1]: import inspect
In [2]: import sys
In [3]: sys.setrecursionlimit(50) # I'm setting this to 50 to make the traceback shorter.
In [4]: stack_log = []
In [5]: def recur():
stack_log.append(len(inspect.stack()))
recur()
...:
In [6]: recur()
We get the traceback (note: there is no need to read it now, so just move forward to the next section).
---------------------------------------------------------------------------
RecursionError Traceback (most recent call last)
<ipython-input-6-45136123341b> in <module>()
----> 1 recur()
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
----> 2 stack_log.append(len(inspect.stack()))
3 recur()
4
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in stack(context)
1462 def stack(context=1):
1463 """Return a list of records for the stack above the caller's frame."""
-> 1464 return getouterframes(sys._getframe(1), context)
1465
1466 def trace(context=1):
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in getouterframes(frame, context)
1439 framelist = []
1440 while frame:
-> 1441 frameinfo = (frame,) + getframeinfo(frame, context)
1442 framelist.append(FrameInfo(*frameinfo))
1443 frame = frame.f_back
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in getframeinfo(frame, context)
1412 start = lineno - 1 - context//2
1413 try:
-> 1414 lines, lnum = findsource(frame)
1415 except OSError:
1416 lines = index = None
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in findsource(object)
742 is raised if the source code cannot be retrieved."""
743
--> 744 file = getsourcefile(object)
745 if file:
746 # Invalidate cache if needed.
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in getsourcefile(object)
670 return filename
671 # only return a non-existent filename if the module has a PEP 302 loader
--> 672 if getattr(getmodule(object, filename), '__loader__', None) is not None:
673 return filename
674 # or it is in the linecache
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in getmodule(object, _filename)
699 # Try the cache again with the absolute file name
700 try:
--> 701 file = getabsfile(object, _filename)
702 except TypeError:
703 return None
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in getabsfile(object, _filename)
683 if _filename is None:
684 _filename = getsourcefile(object) or getfile(object)
--> 685 return os.path.normcase(os.path.abspath(_filename))
686
687 modulesbyfile = {}
/Users/ilia/.venvs/py3/bin/../lib/python3.5/posixpath.py in abspath(path)
355 def abspath(path):
356 """Return an absolute path."""
--> 357 if not isabs(path):
358 if isinstance(path, bytes):
359 cwd = os.getcwdb()
/Users/ilia/.venvs/py3/bin/../lib/python3.5/posixpath.py in isabs(s)
61 def isabs(s):
62 """Test whether a path is absolute"""
---> 63 sep = _get_sep(s)
64 return s.startswith(sep)
65
RecursionError: maximum recursion depth exceeded
What’s with the stack log?
In [7]: stack_log[-1]
Out[7]: 39
Okay, we have 11 missing frames. Now, scroll down the traceback to the last recur call, i.e.
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
2 stack_log.append(len(inspect.stack()))
----> 3 recur()
4
<ipython-input-5-643b16f38b2e> in recur()
1 def recur():
----> 2 stack_log.append(len(inspect.stack()))
3 recur()
4
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in stack(context)
1462 def stack(context=1):
1463 """Return a list of records for the stack above the caller's frame."""
-> 1464 return getouterframes(sys._getframe(1), context)
1465
1466 def trace(context=1):
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in getouterframes(frame, context)
1439 framelist = []
1440 while frame:
-> 1441 frameinfo = (frame,) + getframeinfo(frame, context)
1442 framelist.append(FrameInfo(*frameinfo))
1443 frame = frame.f_back
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in getframeinfo(frame, context)
1412 start = lineno - 1 - context//2
1413 try:
-> 1414 lines, lnum = findsource(frame)
1415 except OSError:
1416 lines = index = None
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in findsource(object)
742 is raised if the source code cannot be retrieved."""
743
--> 744 file = getsourcefile(object)
745 if file:
746 # Invalidate cache if needed.
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in getsourcefile(object)
670 return filename
671 # only return a non-existent filename if the module has a PEP 302 loader
--> 672 if getattr(getmodule(object, filename), '__loader__', None) is not None:
673 return filename
674 # or it is in the linecache
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in getmodule(object, _filename)
699 # Try the cache again with the absolute file name
700 try:
--> 701 file = getabsfile(object, _filename)
702 except TypeError:
703 return None
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/inspect.py in getabsfile(object, _filename)
683 if _filename is None:
684 _filename = getsourcefile(object) or getfile(object)
--> 685 return os.path.normcase(os.path.abspath(_filename))
686
687 modulesbyfile = {}
/Users/ilia/.venvs/py3/bin/../lib/python3.5/posixpath.py in abspath(path)
355 def abspath(path):
356 """Return an absolute path."""
--> 357 if not isabs(path):
358 if isinstance(path, bytes):
359 cwd = os.getcwdb()
/Users/ilia/.venvs/py3/bin/../lib/python3.5/posixpath.py in isabs(s)
61 def isabs(s):
62 """Test whether a path is absolute"""
---> 63 sep = _get_sep(s)
64 return s.startswith(sep)
65
RecursionError: maximum recursion depth exceeded
And here you are, there are exactly 11 function calls (the arrows on the left), that is 11 frames on the stack that were removed when the exception was raised.