Why does Python have a limit on the number of static blocks that can be nested?
Question:
The number of statically nested blocks in Python is limited to 20.
That is, nesting 19 for loops will be fine (although excessively time consuming; O(n^19) is insane), but nesting 20 will fail with:
SyntaxError: too many statically nested blocks
What is the underlying reason for having such a limit?
Is there a way to increase the limit?
Answers:
See answer here: too many statically nested blocks python
You cannot increase it as it is builtin to the python syntax. The limit applies to any kind of code stack (exceptions, loops, etc.) and is a decision by the designers (presumably to keep memory usage reasonable).
One weird thing is that here: https://github.com/python/cpython/blob/6f0eb93183519024cb360162bdd81b9faec97ba6/Include/code.h#L95 it says 20 is the max number in a function. But I just tried nesting 23 for loops, not inside a function, and you still get the error.
This has to do with the blockstack, which is a stack of the byte code addresses, and is used to execute code blocks such as loops and exceptions.
It just so happens that a version of C (older than C99) had set this limit to 20, and since the CPython interpreter is built with C, the same convention has been followed:
#define CO_MAXBLOCKS 20 /* Max static block nesting within a function */
The constant 20 seems to be set out of convention, and nothing more.
[Links courtesy Christian Dean.]
Why is the limit 20?
If the argument of convention isn’t convincing, then take a look at The Zen of Python:
In [4]: import this
The Zen of Python, by Tim Peters
...
Flat is better than nested.
...
How can you increase this value?
Since this value is a hardcoded constant, the only way to change it to effect in your programs is to rebuild your python distribution and run your script on the new build.
-
Download the cpython source code from github
-
Navigate to cpython/Include/code.h
-
Change the value of CO_MAXBLOCKS to anything greater than 20
-
Recompile Python (disable tests, they’ll complain)
This limit not only applies to for loops, but to all other control flow blocks as well. The limit for the number of nested control flow blocks is defined inside of code.h with a constant named CO_MAXBLOCKS:
#define CO_MAXBLOCKS 20 /* Max static block nesting within a function */
This constant is used to set the maximum size for the stack Python uses to execute exceptions and loops named blockstack. This limit is imposed upon all frame objects and is shown in frameobject.h:
int blockstack[CO_MAXBLOCKS]; /* Walking the 'finally' blocks */
The most likely reason for this limit is to keep memory usage at a sane level when executing nested blocks. It’s probably similar to the limit Python imposes on recursive calls. This limit can be seen being enforced in compile.c:
if (c->u->u_nfblocks >= CO_MAXBLOCKS) {
PyErr_SetString(PyExc_SyntaxError,
"too many statically nested blocks");
return 0;
}
A more concrete answer as to why Python has this specfic limit and why they cannot get rid of it, was given by Michael Hudson in a 2004 Python mailing list letter:
Spot on. This has to do with the ‘blockstack’, very much an internal
detail to Python’s implementation. We’d like to get rid of it (not
because we want to let people write code with more than 20 nested for
loops 🙂 but it’s not especially easy (finally: blocks are the
biggest problem).
Note that in Python 2.6 and lower, breaking the maximum number of nested loops would’ve cause a SystemError not a SyntaxError. This was changed however in Python 3 and back-patched to Python 2.7 so a SyntaxError would be raised instead. This was documented in #issue 27514:
Issue #27514: Make having too many statically nested blocks a SyntaxError
instead of SystemError.
The reason for this change in exception types was given by Serhiy Storchaka :
[…] SystemError is not an exception that should be raised. SystemError is for errors that can’t be occurred in normal case. It should only be caused by incorrect use of C API or hacking Python internals. I think SyntaxError is more appropriate in this case […].
The number of statically nested blocks in Python is limited to 20.
That is, nesting 19 for loops will be fine (although excessively time consuming; O(n^19) is insane), but nesting 20 will fail with:
SyntaxError: too many statically nested blocks
What is the underlying reason for having such a limit?
Is there a way to increase the limit?
See answer here: too many statically nested blocks python
You cannot increase it as it is builtin to the python syntax. The limit applies to any kind of code stack (exceptions, loops, etc.) and is a decision by the designers (presumably to keep memory usage reasonable).
One weird thing is that here: https://github.com/python/cpython/blob/6f0eb93183519024cb360162bdd81b9faec97ba6/Include/code.h#L95 it says 20 is the max number in a function. But I just tried nesting 23 for loops, not inside a function, and you still get the error.
This has to do with the blockstack, which is a stack of the byte code addresses, and is used to execute code blocks such as loops and exceptions.
It just so happens that a version of C (older than C99) had set this limit to 20, and since the CPython interpreter is built with C, the same convention has been followed:
#define CO_MAXBLOCKS 20 /* Max static block nesting within a function */
The constant 20 seems to be set out of convention, and nothing more.
[Links courtesy Christian Dean.]
Why is the limit 20?
If the argument of convention isn’t convincing, then take a look at The Zen of Python:
In [4]: import this
The Zen of Python, by Tim Peters
...
Flat is better than nested.
...
How can you increase this value?
Since this value is a hardcoded constant, the only way to change it to effect in your programs is to rebuild your python distribution and run your script on the new build.
-
Download the cpython source code from github
-
Navigate to
cpython/Include/code.h -
Change the value of
CO_MAXBLOCKSto anything greater than 20 -
Recompile Python (disable tests, they’ll complain)
This limit not only applies to for loops, but to all other control flow blocks as well. The limit for the number of nested control flow blocks is defined inside of code.h with a constant named CO_MAXBLOCKS:
#define CO_MAXBLOCKS 20 /* Max static block nesting within a function */
This constant is used to set the maximum size for the stack Python uses to execute exceptions and loops named blockstack. This limit is imposed upon all frame objects and is shown in frameobject.h:
int blockstack[CO_MAXBLOCKS]; /* Walking the 'finally' blocks */
The most likely reason for this limit is to keep memory usage at a sane level when executing nested blocks. It’s probably similar to the limit Python imposes on recursive calls. This limit can be seen being enforced in compile.c:
if (c->u->u_nfblocks >= CO_MAXBLOCKS) {
PyErr_SetString(PyExc_SyntaxError,
"too many statically nested blocks");
return 0;
}
A more concrete answer as to why Python has this specfic limit and why they cannot get rid of it, was given by Michael Hudson in a 2004 Python mailing list letter:
Spot on. This has to do with the ‘blockstack’, very much an internal
detail to Python’s implementation. We’d like to get rid of it (not
because we want to let people write code with more than 20 nested for
loops 🙂 but it’s not especially easy (finally: blocks are the
biggest problem).
Note that in Python 2.6 and lower, breaking the maximum number of nested loops would’ve cause a SystemError not a SyntaxError. This was changed however in Python 3 and back-patched to Python 2.7 so a SyntaxError would be raised instead. This was documented in #issue 27514:
Issue #27514: Make having too many statically nested blocks a SyntaxError
instead of SystemError.
The reason for this change in exception types was given by Serhiy Storchaka :
[…] SystemError is not an exception that should be raised. SystemError is for errors that can’t be occurred in normal case. It should only be caused by incorrect use of C API or hacking Python internals. I think SyntaxError is more appropriate in this case […].